Community Resilience and Recovery Initiative: Final Evaluation Report

"

Acronyms

The following acronyms are mentioned in this report.

 

Executive Summary

In the fall of 2010, the Substance Abuse and Mental Health Services Administration (SAMHSA) launched the Community Resilience and Recovery Initiative (CRRI). CRRI was a multi-level, place-based demonstration project aimed at helping grantee communities cope with the ongoing behavioral health effects of the Great Recession. SAMHSA funded three grants based on applications submitted in response to its Request for Applications (RFA): Union City, New Jersey; Fall River, Massachusetts; and Lorain, Ohio. Each applicant was awarded $1.4 million a year for 2 years to improve the coordination and availability of behavioral health services in their respective communities. The RFA anticipated that funding would be available for up to 4 years, but ultimately only 2 years of funds were available. Grantees then operated for up to 1 year more on carryover monies. Required activities included social marketing efforts, community-wide screenings, provision of brief interventions (such as motivational interviewing), and referrals to more intensive services, as needed. The initiatives also required grantees to work in collaboration with various social service agencies in their communities, including employment and job training agencies, mental health service providers, and agencies and organizations that provide services to combat substance use disorders.

To assess the implementation and potential success of these grants, the Office of the Assistant Secretary of Planning and Evaluation within the U.S. Department of Health and Human Services awarded Westat a contract to evaluate the initiative throughout the program's duration. The objectives of the evaluation were threefold: First, to describe the characteristics of grantee implementation processes. In order to achieve this objective, Westat conducted two-person site visits to each grantee community in the fall of 2011, 2012, and 2013. In each site the evaluation team conducted in-depth interviews with key project staff, staff from partner agencies, and service recipients. An important finding from the site visits was that each grantee made significant adjustments to the original program design in order to meet their community's unique needs. In Union City, for example, the emphasis of the program was on providing in-school substance use services to ensure that young people caught using drugs or alcohol would receive appropriate treatment and be able to complete their high school education on time. In Lorain, the director of the employment program paid particular attention to the city's African American community, which had been hit by the Great Recession, but also had been disproportionately affected by previous economic downturns. Finally, Fall River used a case management approach to meet its clients' economic and behavioral health needs. This service delivery model allowed clients to establish 6-month relationships with their case managers, which resulted in excellent recordkeeping and strong outcomes. However, the model was much more intensive than that envisioned in the original RFA.

The second objective of the Westat evaluation was to report on the client outcomes achieved by each of the grantees. Client information was recorded by grantees in the Services Accountability Improvement System, the data system developed by SAMHSA to meet the Government Performance and Results Act requirements. Each year of the evaluation, SAMHSA sent Westat a set of de-identified, client-level data for all three grantees. Westat staff then analyzed the dataset for descriptive information about enrolled clients, overall client outcomes, and the effectiveness of several grantee programs. Overall, employment and behavioral health data indicated that clients were doing much better 6 months after enrolling in the program than they were at the point of intake. In each community, more clients were employed at follow-up than at intake; substance use and abuse had decreased 6 months after program enrollment; and clients reported fewer symptoms of depression and anxiety at follow-up than when they enrolled in the programs. Although the study design does not allow us to claim that the programs were responsible for these improvements,¹ the data are encouraging.

Third and final objective of the evaluation was to assess the extent to which this place-based initiative was able to improve community-level resilience in the face of adverse economic circumstances. Grantees were to conduct surveys in their communities each year to get measures of residents' sense of well-being, as well as collect key community indicators (e.g., number of domestic violence incidents, number of alcohol-related or drug-related hospitalizations) that would allow Westat to assess community-level change over time. With the exception of Union City, grantees struggled to implement community surveys and obtain consistent and reliable community-level indicators. As a consequence, we were unable to meet the third objective of the evaluation.

This project resulted in several important lessons learned. First, all three grantees noted the value of linking behavioral health and employment services in their communities. Interviewees reported that for many of their clients, seeking employment assistance or job training support is less stigmatizing than asking for help for depression, anxiety, or a substance use disorder. Employment services thus functioned as a safe gateway for clients in need of additional assistance. In addition, interviewees said the CRRI initiative alerted the service providers to the potential behavioral health sequelae from losing a job. Providers in all three sites reportedly had not really thought about this connection, and the projects opened their eyes to the potential emotional distress that can results from economic difficulties. The work-behavioral health connection thus appears to have significant potential to make a difference for both help-seekers as well as those providing the assistance and perhaps merits additional exploration by SAMHSA in other grants.

Second, these projects pointed out the value of having sufficient flexibility built into the grants so that programs can be adapted to a community's specific needs. Each of the three CRRI project directors understood the intentions of the grants and incorporated the fundamentals of the RFA (e.g., screening, brief interventions, referrals) into their programs. But each also understood the idiosyncrasies of their communities that required they take a "theme-and-variations" approach to the projects rather than a "cookie cutter" one. Local knowledge allows these and similar initiatives to make a difference in the community and reinforces SAMHSA's philosophy that local communities, rather than outside entities, are best suited to develop solutions to local challenges. Unfortunately, this local variation made it difficult to evaluate the program as an intervention across sites.

Finally, the CRRI initiative was fast-moving and required the grantees to undertake several new activities (e.g., development of a media campaign, creating community partnerships for screening) and begin enrolling clients in their programs within 4 months of the contracts being awarded. Despite the steep learning curve, each of the three grantees did a remarkable job bringing their programs online within or near to the required timeframe. There were some small missteps during that run-up (e.g., enrolling clients at the point of referral, rather than when the client arrived for services), but project directors provided excellent leadership and problem-solving to bring the programs past these hurdles. One lasting challenge, however, was having the grantees conduct the data collection for the community evaluation. Even though each hired an outside evaluator, grantees were oriented towards service delivery, not the evaluation of those services. Having the grantees start from ground zero to develop a community survey instrument was perhaps one requirement too many in an already ambitious initiative. It may be worth considering both the cost and data quality associated with having site-based evaluators and assess whether hiring an outside evaluation firm would be less costly -- or at least cost neutral -- and result in a higher quality assessment of the programs.

1. Background and Introduction

The Great Recession is a term covering the period from December 2007 until June 2009 that saw a significant shrinking of the American economy. The collapse of the housing market, which had grown significantly on a foundation of marginal loans, triggered the downward spiral of financial markets, consumer spending, and unemployment. By the end of this 16-month period, it was estimated that the labor market lost more than 8.4 million jobs.² In December 2007, the national unemployment rate stood at 5.0 percent; by June 2009, it had risen to 9.5 percent and continued to climb for a few more months; by October 2009 the unemployment rate was up to 10.0 percent nationally.³ Although the last 4.5 years have seen significant economic recovery, the labor market has not rebounded to its pre-recession levels: In December 2013, the national unemployment rate still stood at 6.7 percent.⁴

In response to the recession and the slow recovery process, in 2010, the Substance Abuse and Mental Health Services Administration (SAMHSA) launched the Community Resilience and Recovery Initiative (CRRI). CRRI was a multi-level, place-based demonstration project aimed at helping grantee communities cope with the ongoing behavioral health effects of the Great Recession. SAMHSA funded three grants based on applications submitted in response to its Request for Applications (RFA): Union City, New Jersey; Fall River, Massachusetts; and Lorain, Ohio. Each applicant was awarded up to $1.4 million a year for up to 4 years to improve the coordination and availability of behavioral health services in their respective communities. More details about specific grantee requirements are set forth in the following chapters.

In addition to these implementation grants, the Office of the Assistant Secretary of Planning and Evaluation (ASPE) within the U.S. Department of Health and Human Services awarded Westat a contract to evaluate the initiative throughout its duration. The objectives of the evaluation were threefold: First, to describe the characteristics of grantee implementation processes, including social marketing efforts, screening procedures, brief interventions, and referrals to services; second, to report on the individual-level outcomes achieved by each of the grantees; and finally, to assess the extent to which this place-based initiative was able to improve community-level resilience in the face of adverse economic circumstances. An additional contract to support the initiative was established between SAMHSA and the National Association of State Mental Health Program Directors, whose staff conducted background research for the grantees on various evidence-based programs (e.g., suicide prevention, employment training and support). SAMHSA also contracted with Gallup to provide technical assistance to grantees as they developed their community media campaigns.

Although there was the potential for grants to extend for up to 4 years, funding was only available for 2 years. This significantly changed the trajectory of grantees' program implementation efforts, as well as the evaluation design.

This report summarizes our evaluation findings for this abbreviated, but important initiative. In the next chapter, we provide an overview of the RFA, which sets forth the objectives of the grant as well as the responsibilities of the grantees. We also provide a brief description of each of the grantee communities, emphasizing those community characteristics that made each awardee an apt candidate for this endeavor. Chapter 1 concludes with a discussion of how funding for 2 years instead of 4 years impacted both the implementation by the grantees as well as the Westat evaluation design. Chapter 3 focuses on the grantees' efforts and is divided into a separate section for each grantee community. Consistent with our evaluation goals, we describe the implementation processes for each grantee and detail their respective accomplishments in their communities. In Chapter 4, the reader will find our analysis of individual-level and community-level outcomes. For the former, we summarize the findings from the Services Accountability Improvement System (SAIS) data received from SAMHSA, including the characteristics of individuals who were enrolled in the CRRI program in each community; outcomes for key indicators, such as substance use disorders, employment, and mental health; and, where feasible, details about program effectiveness. Although the findings do not imply causality, the tables and graphs in this chapter clearly indicate that enrolled clients evinced improvement in both employment and behavioral health indicators over time. This chapter also includes a discussion about the community-level data and some of the challenges associated with that analysis. Chapter 5 provides a brief summary of the evaluation findings and implications for future place-based initiatives.

2. Overview of the Study

The RFA for the CRRI was released in the spring of 2010 with the following stated objective:

The purpose of this place-based initiative is to improve behavioral health outcomes through enhanced coordination and evidence-based health promotion, illness prevention, treatment, and recovery support services in communities affected by the recent economic downturn.

Eligible applicants were to implement a four-level initiative in collaboration with other service providers in their communities. Level I was to develop a media campaign to reduce the stigma about seeking help for behavioral health services and to inform residents about available resources in the community. Level II required awardees to identify and implement evidence-based services to prevent the emergence of behavioral health issues. For Level III, grantees were to develop and implement a community-wide plan for screening residents for behavioral health issues, providing evidence-based brief interventions for emerging challenges, and referring individuals with more profound issues to service providers in the community who could provide more intensive treatment, as needed. The implementation of this Screening, Brief Intervention, Referral and Treatment (SBIRT) model aimed explicitly to reduce the negative behavioral health effects of the economic downturn. And the final level, Level IV, required grantees to have a plan of intervention for residents in behavioral health crises.

In addition to the above components, awardees were required to collect and report data on enrolled clients so that SAMHSA could meet its Government Performance and Results Act (GPRA) requirements. Ultimately grantees recorded this information in SAMHSA's SAIS data system. Finally, grantees were required to participate in the evaluation of the initiative. This included participating in the site visits conducted by Westat, such as facilitating the visits and participating in any qualitative interviews conducted during the visits. In addition, grantees were required to collect and report selected community-level outcomes, such as rates of domestic violence, reports of child maltreatment, and substance abuse-related hospitalizations. Grantees also had to conduct a community survey to assess the effectiveness of their media campaigns. These outcomes were critical to determining if the initiative was having the intended impact on the whole community. Finally, grantees were required to report on various program-level outcomes, such as the number of individuals screened, number of individuals receiving various services (e.g., employment support, substance use disorder treatment, mental health treatment, parenting classes), and the number of collaborations established with other providers in the community.

Grantee Communities

Three communities that had been particularly hard-hit by the Great Recession submitted applications for and were awarded the funds to implement the CRRI program. One was Union City, New Jersey, a compact (1.27 square miles) urban community located just across the Hudson River from Manhattan. Its population of 66,500 is largely Hispanic (85 percent). Many people in Union City work in the manufacturing and service industries, both of which took a major hit in the recent recession. Union City experienced a rise in unemployment from 9 percent in 2007 to 14 percent in 2009. In 2010 when the initiative began, the unemployment rate had recovered from its 2009 peak, but was still above pre-recession levels at 10.4 percent. In its application, the Mayor's Office partnered with North Hudson Community Action Corporation (NHCAC) and proposed to implement three evidence-based prevention programs. One was the JOBS Project, a program out of the University of Michigan that focused on providing job-seeking skills and mitigating feelings of depression or anxiety among individuals who were out of work. In addition to this focus on displaced workers, Union City proposed to combat the challenges of youth substance use disorders by implementing the Strengthening Families Program and the Strengthening Families Program for Parents and Youth 10-14. Because of significant challenges with substance use in the city's schools, eligible youth and families would be recruited through the school district. All three programs were to be translated to meet the needs of the city's Spanish-speaking population.

A second grantee was Fall River, Massachusetts. This community, located near the border with Rhode Island, was once one of the leading textile manufacturing cities in the United States. Throughout the 19th and early 20th centuries, the city remained vibrant as the types of industries changed with the transformations in the American economy. Shortly after World War II, however, Fall River's fortunes began to decline. Indeed, over the past quarter century it suffered from the continued loss of positions in both the manufacturing and service sectors, a loss that was exacerbated by the Great Recession. In 2007, the city's unemployment rate was 8.3 percent; by 2010, the unemployment rate was in double digits at 15.8 percent⁵ and an estimated 20 percent of residents were living in poverty (Census Bureau, 2010). With the depressed economy came a host of behavioral health challenges for city residents. Compared with the state as a whole, Fall River adults reported higher rates of depression and rates of injection drug use (primarily heroin).

In its application, the Mayor's Office partnered with a long-standing substance use disorder service provider in the city, SSTAR, to develop and implement the four-level intervention spelled out in the RFA. Ultimately, SSTAR proposed to use a 6-month case management approach to meet the behavioral health needs of individuals who screened positive for depression, anxiety, or problematic substance use. Other evidence-based interventions including Parenting Wisely and The Incredible Years, two family strengthening programs that have a strong research base; the ARISE intervention, a substance use intervention approach that was successfully tested by the National Institute on Drug Abuse; and a Job Club, an employment training and support group for displaced workers.

The third funded application was from Lorain, Ohio, a town of about 70,000 residents located on the shores of Lake Erie, approximately 30 miles west of Cleveland. The area's experience with economic decline had spanned decades and was largely related to the falling fortunes of the auto industry and, more generally, manufacturing. The negative impacts of the Great Recession were particularly egregious, exacerbating the social and economic distress of Lorain's citizens. In December 2007, the unemployment rate among city residents was 7.3 percent; by February 2009, that figure had soared to 12.5 percent.⁶ Selected city data compiled in 2008⁷ reflected a poverty rate among community residents that was over 25 percent; a local crime rate that surpassed the national average; and a local median income that was $13,000 below the median income for Ohio residents. The recession also was felt at the state level, resulting in budget cuts to an array of service organizations, including those that deliver health care, mental health care, or substance use disorder treatment services.

It was within this context that the Mayor's Office, working in partnership with the Alcohol and Drug Addiction Services (ADAS) Board of Lorain County, submitted its application for the CRRI initiative. Their proposed program, Partners Realistically Integrating Durable Empowerment (PRIDE), included several evidence-based strategies, including the Strengthening Families Program, a Spanish-language version of the Strengthening Families Program for Youth 10-14, and the same JOBS Project that was identified by Union City as an evidence-based intervention for displaced workers. Lorain was unlike the other two sites in that the application was submitted by a board, not a service provider. Thus the successful implementation of PRIDE required a strong Policy Steering Committee and committed partnerships among service providers in the city.

Westat's Evaluation

Westat was awarded a contract from ASPE with funding from SAMHSA to evaluate the implementation of the initiative in all three communities. The objectives of the evaluation were threefold: First, to describe the characteristics of grantee implementation processes, including social marketing efforts, screening procedures, provision of brief interventions, and referrals to more intensive services, as needed. This information was to come primarily from annual site visits to the grantee communities during which time Westat evaluators would interview project staff, Steering Committee members, community partners, and enrolled clients. The second evaluation objective was to report on the individual-level outcomes achieved by each of the grantees. As noted previously, grantees were required to record information about enrolled clients' characteristics and services in the SAIS data system. SAMHSA analysts cleaned these data and submitted them annually to Westat for subsequent analysis. Finally, Westat was to assess the extent to which the CRRI initiative was able to improve community-level resilience in the face of adverse economic circumstances. Data for this third objective were to be collected by grantees through local contacts (e.g., local emergency room data) and/or administrative data, and through the annual administration of a community-level survey.

Assessing the extent to which the CRRI initiative was able to improve community-level resilience proved problematic. The primary challenge that prevented Westat from meeting this objective was the requirement spelled out in the RFA that grantees were responsible for collecting the needed community-level data elements. Although grantees were required to hire their own evaluators to assess their program implementation and progress, evaluators had different ideas about what their responsibilities were for the community-level component. This challenge is discussed in more detail in Section 4.5.

Loss of Funding for CRRI

As noted previously, the original RFA indicated that monies could be available for up to 4 years for the CRRI programs. Ultimately, however, grantees received just 2 years of funds to implement their programs. This change in anticipated funding did more than just halve the time for implementation; it also shifted grantees' focus from program development to sustainability. This was a significant change given that grantees were only 1 year into the initiative and thus still troubleshooting various aspects of implementing their proposed programs. It also generated uncertainty for many staff who had anticipated being employed with the program for up to 4 years. It is a testament to the grantees' commitment to their communities that they continued to enroll and serve clients throughout the second year of funding and, indeed, during the time periods when they were operating with carryover monies. In addition, and as we describe in the site summaries in the next chapter, each grantee was able to sustain at least one or more component of its program. These are incredible accomplishments under any circumstances, much less the ones faced by these three grantees. It should be recognized, however, that by the start of the second year of funding, grantees' activities had shifted from their intended focus.

Another implication of the change in funding is that Westat's evaluation design shifted as well. The two remaining site visits that were planned to each of the three grantee communities shifted from focusing on implementation processes to providing an assessment of sustainability. The first of these two visits, in September 2012, focused on grantees' perceptions of the value of the initiative to their community and what steps they planned to take to sustain the service delivery model when funding ended. The second set of visits occurred in October 2013 after any carryover funds had been expended and projects were closed. Site visitors were to explore what elements of the programs actually had been sustained and what organization(s) in the community had continued with those CRRI project efforts. The deliverable from these visits is included as the site summaries in Chapter 3 of this report, and details grantees' innovations, challenges and accomplishments. Consistent with the original contract modification, Westat produced an outcomes report using SAIS data from the grantees. That report is included as Chapter 4 in this document.

What the reader will find in the next two chapters is documentation of an initiative that appears to have been successful in many facets of its implementation. Site summaries detail grantees' incredible efforts on behalf of their communities, and data on enrolled clients indicates an increase in employment among participants and a decrease in behavioral health symptoms over the course of their participation in the program. Although the RFA specified the service delivery approach and target population of this initiative, the reader will find that grantees created variations on the specified "theme" consonant with the perceived needs of their communities. These innovations resulted in three non-comparable programs for the evaluation, but reflect the wisdom of SAMHSA's philosophy that local problems require local solutions.

3. Site Reports

Three locations were selected to implement the CRRI and all shared the characteristics of communities experiencing long-standing economic distress. Nevertheless, they were three distinct communities whose history and needs varied significantly. Through these grants, the cities were asked to implement a common screening program but were allowed to tailor the program to meet their unique circumstances. This resulted in what we can best describe as a "theme and variations" approach. Thematically, the sites shared similar strategies for managing staff, using the SBIRT model, conducting professional training, and collaborating with key community partners. In term of variations, the sites used different service delivery strategies and focused on very different populations. In this chapter, we present a site report for each of the grantees. These reports demonstrate the grantees' adherence to the basic concept of the CRRI grant, but also highlight the innovations that each brought to their effort.

3.1. Project ASSIST: Fall River, Massachusetts

The Fall River CRRI Project, Project Assisting, Serving, Supporting in Stressful Times (ASSIST), was implemented by a large non-profit service delivery organization, SSTAR, that has been in the community for more than 35 years. In addition to providing behavioral health care and substance use disorder services, SSTAR operates a Federally Qualified Health Center (FQHC) that provides medical services to Fall River residents of all ages. Project ASSIST ably demonstrated the value of having the grant come through the Mayor's Office, which served as the fulcrum of community collaboration. In addition, the project's outreach and screening strategies, coupled with a 6-month case management approach, allowed over 250 residents to receive employment and behavioral health supports. Although the case management model could not be sustained by SSTAR following the completion of CRRI, there are many components of the grant that the agency will carry through into the future.

Staffing Strategies

Project ASSIST was successful in conducting tactical hiring practices for direct services staff. The leadership selected for Project ASSIST included a highly trained and experienced substance use disorder professional as the grant coordinator and an evaluator who also had extensive experience assessing program implementation. Eight staff took on the role of case manager, which was a position not typically used by the agency. The individuals hired were familiar with SSTAR and SBIRT concepts and practices. Some case managers were promoted from within SSTAR, and others were hired from outside the agency. These staff came from diverse backgrounds and had bachelor's degrees. Over the summer months, the grant allowed the project to hire seven part-time college interns to help with outreach activities in the community. In addition, grant funds allowed for the hiring of an administrative professional to assist the project. Some of the staff were co-located in the Career Center and High School.

Staff received extensive training and professional development in screening and brief intervention techniques. Staff from Project ASSIST and additional staff representing community partners were trained on a variety of new service delivery methods. These included a 4-day training on a family strengthening program called The Incredible Years; a training on Seeking Safety, and a number of other trainings specific to the grant, including Motivational Interviewing, FRAMES, SBIRT, and the ARISE Intervention. Some staff also participated in trainings on Parenting Wisely, the administration of GAIN, Non-violent Crisis Intervention, CADAC Prep Course, Suicide Prevention, and Cognitive Behavioral Therapy (CBT). Project staff also received orientation to the evaluation elements of this project, including the screening tools to be used, confidential data collection protocol, and data entry through the SAMHSA SAIS system. Training was also offered by the Gallup organization with the focus of improving marketing of the program in general. During the Year 3 site visit by Westat, interviewees reported that the multiple staff training opportunities strengthened the overall staff skill base, which continues to be an asset to the agency as it works to meet its clients' needs.

Community Partnerships

When the grant started in 2010, SSTAR already had a long history of successful community partnerships. Nevertheless, the specific requirements of CRRI fostered new connections throughout the community that were essential to the success of the project. The linkage with Mayor William A. Flannigan's office, for example, provided specific guidance and resources to the effort along with entry to other government agencies and local and federal leaders. A project Steering Committee was convened by the Mayor, which resulted in new links with Bristol Community College and job services. Collaborations also were developed with other behavioral health and social service organizations, neighborhood associations, juvenile courts, local hospitals, and veterans' groups. In addition, the actual establishment of outreach and screening strategies and physically being present at various sites built new relationships with a wide range of agencies and created a more public face for the issues of behavioral health community-wide.

Such collaborative efforts on CRRI-specific tasks also altered and improved community members' perceptions of SSTAR as a comprehensive service provider. Prior to the grant, SSTAR reportedly was viewed primarily a substance use disorder service provider and detox agency. But through CRRI, the community-wide perception and understanding of SSTAR as a resource for substance use disorders, mental health, medical services, and employment services has broadened. In addition, because SSTAR screening and outreach staff worked on-site with staff from other organizations, these interactions provided opportunities to educate other agency staff on the relationship between mental health, substance use, and economic stress.

Outreach and Screening Strategies

Outreach through information dissemination and health screenings in new locations throughout the community was an important new educational approach for Fall River. The locations used to host the screenings encompassed a wide range of community sites including the Career Center, the community college, the Family Health Center, the Government Center, Veterans Center, juvenile courts and community locations such as the farmer's market, neighborhood associations, and special community events like "Fun Days" designed to draw in a wide variety of residents.

Many residents who were screened in the moderate risk range and were unemployed or underemployed were eligible to receive services and were referred into the case management program and for behavioral and substance use program interventions. Project staff found that many of these enrollees had never received behavioral health services, and so were unaware of the resources available for depression and substance use disorders. One of the unique aspects of this CRRI site is that the city represents cultural and language diversity with a significant Portuguese population, (12 percent of those screened were Portuguese compared to 6 percent Black, 6 percent Latino).

• Marketing and Outreach to Community: SSTAR staff developed new and significant marketing skills as a result of their participation in the CRRI project. Marketing efforts included the development of print media items, such as brochures, flyers, newsletters, and advertisement in other organizational partners' newsletters to publicize the project. Staff also became proficient in the use of a variety of social media as they created Facebook, You Tube and Twitter accounts which were linked and heavily networked through Facebook. They also completed seven 30-second public service announcements and two 60-second commercials for Internet use. Other marketing materials developed by the communication team included one longer (8 minute) video detailing Project ASSIST, a running project blog on MyFallRiver.org, and a blog called Changing Times in conjunction with the local newspaper.

• Community Screening: The first screening site was STARR's Family Health Care Center, a FQHC. Further screening was conducted at a wide range of agencies and community sites as noted above. Staff also coordinated screening at community events in high poverty areas, and at two shopping malls.

One of the key accomplishments noted by the project staff was using SBIRT to screen community residents for behavioral health issues. During the screening process, Fall River residents were screened for substance use, depression, and anxiety by using the PHQ-9, CAGE-AID and the GAD-7. Such widespread screening was new for SSTAR, although project staff were incredibly successful at it. In fact, the project goal was to screen 5 percent of the city's population; by the time screening came to a close, staff had screened 5,808 persons which is 6.3 percent of the population. Importantly, interviewees noted during the Year 3 site visit that the data from screenings will be a valuable resource for future planning.

Project staff reported that the combination of the marketing, the outreach and screening activities initiated discussions with community residents and they feel this has helped to mitigate the stigma of seeking help for mental health and substance use issues. The screening revealed a larger percentage of respondents who noted a need to "refer to higher level of care" and more clients with behavioral health issues were identified. Staff report that since the 30 percent of screening was conducted at the SSTAR FQHC this may have yielded more clients with behavioral health issues.

Service Development and Delivery

Project ASSIST used a case management approach, which was a new service delivery model for the sponsor agency, SSTAR. Each client enrolled in ASSIST was assigned to a case manager who worked with him/her as much or as little as needed over a 6-month period. Project leadership noted the value of the case managers to the clients:

That connection with a caring individual has made a big difference to the participants who often feel isolated while unemployed. They help the participants secure health insurance, food stamps, fuel assistance, along with whatever needs exist. Many of the people we are enrolling have never had to access services in the past. They often don't know where to begin or what is available to them.

Expanding services to include job support was also an important addition for the program, not simply for the clients, but also for the staff. Prior to CRRI, the staff were not aware of the depth of emotional issues that individuals and families face during times of job loss. The entire process of working with various job and career resources in the community allowed ASSIST and SSTAR staff to gain a new appreciation for the kinds of adverse life events that may contribute to behavioral health challenges.

During the Great Recession, the local One-Stop agencies experienced an overload of requests and were not able to meet the demands for support. Clients revealed that they perceived employment-related resources as being of primary importance, and sought these services prior to seeking other behavioral health interventions. The ASSIST project configured new job-seeking support resources within the mental health treatment model so there was not a need to choose either one or the other. The newly developed jobs club program at Fall River included some work with the chronically underemployed or unemployed who had long-standing substance use and mental health issues. However, the Fall River project also conducted a strong outreach effort to the target population of those individuals and families affected by recent job loss as outlined in the initial grant proposal.

Finally, the project focused on expanding services to veterans. As noted, staff completed the Seeking Safety training and attended a conference on how to work with military families. One of the case managers began conducting screening in the veterans' office twice per month when veterans came in for their checks, and some were eligible to enroll in Project ASSIST. Other outreach to service members was made at the veterans' drop-in center and with the National Guard. Staff noted that midway through the grant, they began to use the revised GPRA instrument with the additional questions on military service and trauma and symptoms of post-traumatic stress disorder. During this time the project had 30 participants (16 percent) identify their status as veterans. Of the 137 individuals who were asked the trauma-related questions, 61 percent reported a history of trauma, with "nightmares" as the most common symptom identified.

Innovations

Project ASSIST developed several service delivery options that were innovative for the Fall River community:

• Job Services as a Low-Stigma Entry Point into Behavioral Health Supports: As noted, the One-Stop Career Center had been overwhelmed with numerous requests from dislocated workers, many of whom also presented with emotional distress. Having a CRRI representative on-site and available for referral and assistance was cited as invaluable. Job-seeking support also was offered on-site at SSTAR, where project ASSIST offered a Monday morning employment support group. This allowed job seekers to start off the week with a positive, motivating, and encouraging experience. A project staff member with experience with resume writing and interviewing skills began working with most clients enrolled in the program. This Monday morning Job Club continued to be well-attended and valued. Program participants who were experiencing anxiety, depression, and stress from unemployment reported feeling better when focusing on activities related to job attainment. Staff believed that these individuals might not have sought support solely for their emotional distress, but received behavioral health supports by coming into the system through employment-related services.

• Case Management as an Innovation: Case management not only helped clients access needed primary or behavioral health services, but also ensured that clients who were screened were not lost to follow-up. Follow-up using this case management approach allowed this CRRI project to reach over 95 percent of participants for 6-month follow-up. The 6-month relationship with a case manager also reportedly helped the participants feel connected to the program.

• Outreach and Screening as an Enhancement to Service Delivery: Staff reported they were amazed by the success of the outreach component of the grant. The ASSIST tagline, "There is no shame in asking for help" was a message that was repeated throughout the community; interviewees said that many people reported this important message had motivated them to seek assistance. Staff also noted the effectiveness of reaching out to people where they are located; first, rather than waiting for clients to knock on the SSTAR door, the outreach pulled them in naturally. Second, staff at the screening sites turned out to be important referral resources. This network of referring organizations remains in place after the grant. Finally, as suggested earlier, screening such a large proportion of the city's population (over 5 percent) resulted in a dataset that will be used by the agency for future planning.

In addition, screening at the parks and various community events resulted in outreach to new areas of the community, some with concentrations of poverty. Each event drew a huge crowd and thus was a successful screening opportunity. They also offered struggling families a fun, free, safe event for their children.

Consistent with the SBIRT model, staff found screening to be a great way to identify people starting to struggle with mental health issues or substance use and to offer them help before symptoms worsened.

A unique aspect of Project ASSIST was global outreach. STARR has disseminated the success of the grant at local state, national and international venues. For example, the agency CEO, Nancy Paul, presented the CRRI model at a United Nations Conference in Moscow on November 11, 2011.

Challenges

The most significant challenge with the grant reported by staff was the loss of funding in the midst of project implementation. Although Project ASSIST was able to accomplish many improvements through screening efforts, community outreach, and job support, it was not able to focus on all of its plans for this grant. For example, it had hoped to work with the faith-based community, to offer more of the Seeking Safety program to veterans, and to expand agency sustainability in the community. In addition, the shortened funding period placed many of the staff into a state of job-seeking. This reportedly was difficult for staff experiencing the stress of future unemployment while also working with individuals who were unemployed. The loss of funding also resulted in the loss of at least one position and several other staff had to leave the project early to find alternate work for themselves. The loss of financial support also created a morale problem, as remaining staff expressed disappointment with not being able to continue the advanced training that had been part of the grant plans. Finally, the loss of funding put SSTAR in a tricky position related to their rental of a building for project purposes. A multi-year contract had been signed, but with the project ending early, SSTAR had to find other ways to cover the rental of the building. Fortunately they were able to identify an organization to rent part of the space.

Sustainability

During the final Westat site visit, Project ASSIST staff indicated that the Fall River program had been successful in building a collaborative network that will remain as an important community resource. Ongoing work with the Department of Health, Council on Aging and various youth services groups will be maintained going forward. Additionally, through the screening and outreach activities, new partnerships were developed, such as with the one-stop career center. As the community recovers from the Great Recession, and when difficult economic times occur in the future, this network of partners will be instrumental in helping individuals locate employment-related services as well as behavioral health resources. Interviewees attributed much of this success to having the right team of people implementing the project and participating as partners.

Other aspects of the CRRI grant that will have long-range impact and that interviewees believed to be sustainable include:

• Staff awareness of the importance of the Job Club and the collaborations with the career center were both powerful results of this grant. SSTAR will continue links with the job-seeking resources, sharing brochures and using the job support group on site as a valued resource.

• Renewed awareness of the importance of preventive care was identified by the steering committee as a valuable result of this grant. The entire service delivery system will need to identify a way to create a permanent preventive care approach. The project staff are aware that prevention is cost-effective, especially with respect to costs for diabetes, obesity, and depression. In addition, earlier intervention can help reduce the development of chronic, and therefore expensive, psychiatric and substance use problems.

• Political ties that were part of the configuration of the CRRI grant will support sustainability of various aspects of the program. For example, the project's linkage with the Mayor's Office was seen as profoundly important. State Representative Paul has requested a summary of the program to present to the Legislative Delegation and will advocate at the state level for funding for additional support for community-based prevention programs. Such programs will provide high value to the community at minimal cost.

• Continuing some of the project programs will be limited to available financial resources. Project staff were able to identify some insurance carriers that will reimburse for screening activities. Providing additional training for professional staff in some of the newer programs will be dependent upon funding resources.

• Case management as a model will not be continued due to lost funding for positions. Even though this model was perceived as an innovation for the agency, there is not adequate financial support to continue with it.

• Screening through SSTAR staff will not be able to continue as the former Project ASSIST staff have taken different positions. However, at the time of the Year 3 site visit, the project was trying to encourage the hospital to pick up the costs of continued weekend screening.

Lessons Learned

During the final site visit discussions, interviewees emphasized the need to conduct continual outreach and to share mental health and substance program resources with the wider community. Stigma is pervasive and has prevented many individuals in this community from seeking help for mental health and substance abuse treatment. Through outreach and screening, the project was able to introduce the array of available resources to the people who needed support. Outreach and screening also yield valuable information and data about the community, the population and other referral resources. Hence, funding to conduct outreach and screening will result in data the community can use in the future.

The staff and leadership also have identified the importance of remaining flexible and willing to adjust to the ever changing needs of the city's residents. Staff noted that this capacity to adapt was one of the key lessons from participating in this project. They also noted that it is essential to identify financial supports for building strong collaborative partnerships. Many service delivery agencies have limited resources to allow staff to participate in activities that can help to build and maintain collaborative partnerships; however, this is a critical area for the success of community-wide initiatives.

As noted previously, one of the strengths of the project was the requirement to work closely with the Mayor's Office. This strengthened the existing working relationship and provided an excellent structure for managing the grant and for linking the project with critical political support. This level of networking will be considered an essential element for future initiatives.

In working with the newly unemployed, staff found that many potential clients did not know about resources in the community or how they could access these resources. Using a case management approach helped clients develop a personal relationship with an individual who could facilitate connections with supportive resources in the community. Case managers could also ensure that needed services were actually obtained, and that their clients would not get "lost to the system" as they pursued a successful recovery and employment. In addition, Fall River found that creating stronger links between employment resources and behavioral health programs was beneficial for Project ASSIST clients. In addition, they learned that older people who are not technically skilled had significant difficulty trying to navigate the current online job search methods. The Monday Job Club suggested that individuals who are unemployed and seeking work are greatly benefited by having a connection to others who are in a similar situation.

3.2. PRIDE: Lorain, Ohio

The CRRI program in Lorain was called "Partners Realistically Integrating Durable Empowerment," or "Lorain's PRIDE" program. The program was implemented by the Lorain County ADAS Board in collaboration with the City of Lorain's Mayor's Office. Because ADAS is a board and not a service delivery organization, the implementation of the grant differed in some ways from the other two sites. What remains consistent across sites, however, is that parts of the project are being carried on despite the loss of CRRI funding.

Staffing Strategies

Staffing of Lorain's PRIDE program remained fairly consistent throughout the duration of the grant. The executive director of the ADAS board retained a leadership position on PRIDE for the entire project. The project director was hired from out of state by the board and was given responsibility for the employment program (JOBS) and for developing the media strategy. She, too, remained with the project throughout its duration. Mental health, substance use, and other project services were delivered by employees of the partner agencies and reported to the leadership of their respective agencies. Thus, while the organization of service delivery remained with consistent community partners, the reporting structure for those delivering the services was highly decentralized. This created some challenges for the project, which will be discussed later in this report.

Project funding also had a positive impact on staffing within the partner agencies. For example, prior to CRRI, the Lorain County Public Health Department had had to lay off several nurses due to a lack of funds. Thanks to CRRI monies, several public health nurses were brought back from layoffs to provide screening, outreach, and follow-up services to PRIDE clients. The project also benefited from having experienced staff who were well acquainted with the local community and who were bilingual in English and Spanish. Having Spanish-speaking staff available for the duration of the project enhanced the program's ability to provide services to the local Hispanic populations.

An additional strategy used in Lorain to enhance project staffing was to use volunteers to implement some of the project activities. Volunteers were recruited from the community and assisted in a variety of operational functions of the program, including outreach, job support, and job skills development activities, as well as other administrative tasks. Some of the volunteers were former participants in the JOBS program, and were able to benefit from the volunteer work by using it as an "on-the-job training" experience for their resumes. Lorain also hired three masters-level interns to assist in the project implementation. The use of volunteers and interns was a creative and cost-effective way to augment implementation of several aspects of the PRIDE program.

Impact of Staff Training

Training on various procedures was central to the success of the CRRI project, and will continue to enhance the agencies whose staff received any training. Training for some evidence-based programs was provided only to the agency staff involved in that type of service. For example, the SBIRT model was introduced in mental health and substance use programs and was noted by staff as a valuable resource. Behavioral health service providers as well as providers of employment services were all trained on CBT. Substance use providers were trained on the 12-step facilitation therapy (TSF), a new brief program focused on early recovery from alcohol abuse and other drug abuse or addiction problems. Mental health providers were trained on brief couples therapy. As the PRIDE program was being implemented, the screening results indicated the need for other services. As a result, Rx for Change, a smoking cessation program, and medication assisted therapy were added. School staff were trained in a suicide prevention model called Teen Screen. Although the services are still being offered at the high school, there is not widespread confidence that the services are being provided with fidelity to the original model. Finally, with respect to employment-related services, staff participated in the JOBS training offered through the designers of the program from the University of Michigan. Two trainings were offered. The first training was directed toward implementation of the JOBS program within the PRIDE project. The second training was open to other employment service agencies in the community, as well as faith-based organizations. This model has been very successful and is being continued in Lorain through the Urban League.

Finally, PRIDE staff as well as staff from all of the partners on the PRIDE team, including the mental health and substance use agencies, the schools, and the Career Office, received training on the use of Patient Tools, the screening software program that was used in Lorain, and the GPRA reporting requirements.

Community Partnerships

Like all of the CRRI sites, collaboration among community partners was a central aspect of Lorain's PRIDE. The major partners collaborating on this project included the Mayor's Office, the ADAS Board, the Lorain County substance use agency (LACADA), the mental health provider (NORD), as well as the public school system, the local Urban League, the FQHC, and the community college. The local hospital and local employment office were also brought onto the team after the project was funded.

Most of these partners had worked together on other projects and came together to submit the grant. CRRI offered them new ways of working together. For example, screening and outreach offered the partners a new opportunity to collaborate. Agencies worked directly with PRIDE staff during the screening phase, and as they took increased interest in the project, many provided a venue and, in some cases, their own staff to expand screening. The Career Office, for example, provided office space for a PRIDE person on a regular basis and also made referrals for individuals to receive screening. The community college and FQHC also provided space in which screenings were conducted.

The public health nurses were funded by PRIDE to conduct outreach and follow-up and provided a unique public health perspective in the Lorain program. The nurses had been brought back from a layoff and thus were strongly committed to the goals and objectives of the program. Loss of project funding, however, put these individuals' jobs back in jeopardy.

The JOBS program was the only new service initiated by the PRIDE program and implemented by the PRIDE director out of the ADAS Board offices. It was the principal service that introduced clients to the PRIDE program and ultimately linked participants to other community agencies. For example, JOBS clients were given computer access by both Goodwill and Community Action. The Urban League also became a late addition partner and participated in the Year 2 JOBS training. At the time of the Year 3 site visit, the Urban League had conducted a JOBS session and had a staff person assigned to continue the employment services once the grant was officially over.

Finally, and importantly, the PRIDE project director successfully engaged the African American and faith-based communities in the screening, outreach, and employment portions of the program. A large inner city church whose pastor was a member of the ADAS board became an active PRIDE partner and supported the program's efforts by providing the skills of a cadre of volunteers. This collaboration created a strong employment training aspect to the project and also was the link that brought the Urban League into the program. Outreach into the faith community allowed the project to reach the African American and Hispanic communities. El Centro, the Hispanic service agency in Lorain, became an active partner in the first year of the program and developed their own branding for PRIDE in the community.

Outreach and Screening Strategies

The first year of the PRIDE program included an extensive public media campaign with multiple strategies including neighborhood canvassing, mass mailing, literature drops, billboards, newspaper ads publicizing the PRIDE program with messages that normalized help-seeking behavior. The screening committee met every month and reviewed data to determine how program participants learned about the program. As the program progressed, screening data indicated that direct contact with the public health nurses was a more effective outreach strategy than mass media. As a result, PRIDE concentrated its resources for outreach around direct contact. These nurses screened at traditional sites, such as the FQHC, and waiting areas in partner agencies as well as non-traditional sites, including homeless shelters, Catholic Charities, churches, and libraries. New partners, such as faith-based organizations and Goodwill, also provided additional screening sites. In some situations the nurses used blood pressure screening as the way to reach consumers, who perceived less stigma around addressing a physical health problem. As the project progressed the public health nurses expanded screening to public venues like City Hall and grocery stores, as well as community events such as neighborhood fairs. PRIDE also used JOBS graduates and community volunteers for direct contact through neighborhood canvassing and speaking at community events.

Although some of the partners were experienced in screening individuals for service needs within their own agencies, CRRI screening offered a couple of new things. First, all partners were trained to use the Patient Tools software screening program. Second, the screenings were designed to assess a person's comprehensive service needs, not just the person's needs related to an agency's focus. Screening conducted by the mental health agency partner, for example, might result in referrals for family strengthening and job-related services. There were pros and cons to this approach. Because the screener was not tied to any specific provider, it educated those screened about a variety of resources available in Lorain. It also made the staff doing the screening more familiar with services provided by agencies outside their own. The challenges to having such a decentralized process, however, were, first, that the provision of the services was constrained by the capacity of the agency to meet the increased demand. Some services were available immediately and others required a wait. In addition, participants often had to go through a second intake process when they arrived at an agency with their referral. Nevertheless, faith community leaders interviewed during the final Westat site visit indicated that an important contribution of PRIDE was the education of the community about existing resources.

Service Development and Delivery

Lorain's PRIDE offered citizens access to an array of existing services in the community, such as behavioral health services, family strengthening, employment services, and transportation support. Two of the PRIDE services exceeded their grant target in number served, the TSF and the CAST program for teens. The JOBS program served the greatest number of people in Lorain but ended up serving less than the targeted number of clients. Lorain noted a problem with the enrollment of clients into some of the other services offered. For example, enrollment into the Strengthening Families was difficult as many individuals felt the program was too long and less important when in the midst of a job-seeking crisis. This was also noted also in Fall River and Union City. Other problems noted by PRIDE staff included that often clients had a reduction in interest between screening time and the time of enrollment and frequently clients had difficulty showing up for the services despite much effort from program staff to engage them. It is unclear whether this was due to the client discouragement about resources, reluctance to pursue help-seeking, or other difficulties during times of unemployment and stress that prevent adequate follow through and ultimately enrollment.

PRIDE also developed some new services. For example, they encouraged JOBS graduates to volunteer as a way to build their resume through conducting activities such as neighborhood canvassing and outreach in a type of on-the-job training arrangement. This resource for the PRIDE program and opportunity for the job-seeking volunteers would not have been possible without the funding to have PRIDE staff coordinate this effort. In addition, screening in the early months of the program indicated that nicotine addiction was a problem and smoking cessation support was an unmet need. An evidence-based program was introduced by the substance use agency. With the loss of grant funding, however, the only agency with funds to continue to provide the program is Corrections; thus, if a person in Lorain wants to quit smoking, s/he must contact the police department to do so. The loss of grant funding for this effort was called out as a particular loss to the community.

Innovations

Patient-centered screening was identified as an innovation in Lorain. Staff had been familiar with screening tools that aimed to locate consumers for a particular service provider, such as the mental health agency doing depression screening or the health clinic doing blood pressure screening. The PRIDE screening tool offered a comprehensive assessment of the client's needs, which ensured that an individual would learn about an array of available resources for whatever issues s/he might be facing.

As part of the JOBS graduates' employment plan, they were strongly encouraged to take on a volunteer job that would improve their resume for paid employment. Many volunteers for the PRIDE program worked on the neighborhood canvassing, and distribution of program literature. Graduates of the JOBS program also assumed volunteer positions as PRIDE ambassadors. They did some public speaking and represented the program at community events. The Lorain program was the only one of three that made systematic use of the JOBS graduates to both provide on-the-job training and to expand the direct contact outreach into communities.

As noted previously, screening had the added benefit of bringing the attention of service providers to the full range of community resources available. This awareness served to reinforce the collaboration that was part of PRIDE, but will also ensure some level of ongoing collaboration in the future as providers may refer their clients to other agencies in the community.

PRIDE reached out to minority community members through faith-based organizations and endeavored to address certain cultural barriers to help-seeking behavior. PRIDE outreach brought an invitation to the African American community to avail themselves of all the resources available in Lorain.

Challenges

The organizational structure of PRIDE was both strength and a challenge for the program. The ADAS board had a coordinating, but not supervisory, role. This meant that each of the partner agencies was responsible for the implementation and oversight of the programs within its own agency. As a strength, this structure provided increased capability to provide services to the community and strong collaborative relationships as a matter of necessity. As a challenge, it meant there was no centralized authority to guarantee fidelity to the various evidence-based programs. In addition, there was no one who could impose any sanctions or incentives to ensure that evaluation data were submitted in a timely manner. In each of the first two site visits, Westat staff heard of efforts to have a shared data platform; this effort was declared officially dead during the third and final site visit.

By the final year of the project, the Executive Director of the ADAS board and PRIDE project director had each developed a different vision of the PRIDE program in Lorain. The ADAS director was committed to serving the target population of the newly unemployed as specified in the grant. The project director perceived the chronically unemployed as equally impacted by the Great Recession and equally appropriate recipients of PRIDE services. These differences of opinion on allocation of project resources siphoned off some energy from the project.

Like the other two grantees, Lorain's PRIDE found some program elements were not appropriate for the population served. The family strengthening program had difficulty recruiting and retaining participants. Participants reported the program to be too time intensive, particularly for people without jobs who are busy looking for work or cobbling together several part-time positions to bring sufficient income into the household. The family strengthening model thus was dropped midway through program implementation in Lorain.

On the other hand, PRIDE staff observed some positive things about the evidence-based programs. For example, although it was no surprise that mental health services and substance use services carried stigma, what was surprising for staff was that the JOBS program did not. Thus it proved to be an effective portal for some participants to enter into the other previously stigmatized services. This was reinforced during the third site visit, when a pastor of a church serving mostly African Americans noted that through the JOBS program many people learned about and took advantage of other services such as mental health services that had previously been considered "not for us." This was a significant impact of the PRIDE program.

Lorain, like many communities, has a long-standing transportation problem. As a result, many eligible job applicants were disqualified for positions because they could not get to the job site. PRIDE was able to provide transportation on a temporary basis early in the program, but it was not a sustainable service.

It appeared to the evaluation team that the early termination of the grant was particularly challenging in Lorain. For example, the project director was new to the community and thus required additional time to be oriented to both to Lorain and to the project. In addition, the decentralized model required additional time for agencies to either establish new partnerships or adjust their ways of doing business to conform to the CRRI requirements.

Sustainability

Although funds are not available to continue several of the positions funded through the CRRI grant, there are several aspects of PRIDE that will continue to positively impact the community. For example, the project director reported that the PRIDE program has had significant impacts on the service delivery system in Lorain, with participating agencies developing new capabilities from having trained and worked on PRIDE. In addition, community partnerships helped to carry on some of the project tasks. For example, the United Way took over distribution of the updated community resource guide developed by PRIDE. In addition, PRIDE was able to fund some additional JOBS training from the University of Michigan and invited new community partners to attend. The training was provided for existing agencies providing employment services, such as Goodwill, as well as for representatives from the faith community and the Urban League. At the time of the final site visit, the Urban League had scheduled a JOBS session led by a staff person who attended the training.

Lorain is facing several context changes that will impact service delivery in as yet unpredictable ways. The State of Ohio is merging substance use and mental health services agencies. There was concern expressed by the ADAS director that mental health services for teens will disappear in favor of more substance use services. It is not clear how funds will be allocated under the merger. In addition, and at the local level, the Lorain County health department, which was defunded during the Great Recession, will be merged with the Elyria County health department. How that will play out was unknown at the time of the third site visit.

Lessons Learned

The collaborative partnership model of CRRI established a structure for Lorain's PRIDE that brought the mental health, substance use, FQHC, and public schools in closer collaboration and established lines of communication and cooperation that will remain. The screening and direct contact outreach has impacted all the partner agencies and was a strong tool for deepening community knowledge. Similar approaches might yield similar benefits in other communities.

Projects like CRRI can provide critical financial resources for basic services that are entry points for assistance to families in crisis. An important contribution of PRIDE to the Lorain community was having the funds to rehire the team of the public health nurses who had been laid off by the city health department before PRIDE began. These nurses provided the most effective direct contact outreach and recruitment for PRIDE. Unfortunately, near the end of the grant, they were facing further layoffs, but their role in PRIDE was invaluable.

Using JOBS graduates as volunteers had a double benefit for the community. Not only did it expand the outreach component of PRIDE, but it also offered important volunteer experiences for those needing an on-the-job-training experience. This model could benefit other communities. Conversely, some services, such as family strengthening programs, required too much time commitment from people who perceived their primary problem was unemployment or underemployment.

Lorain worked with vulnerable and under-represented persons who faced more chronic hardships related to unemployment. The project focused on empowerment and participatory methods to engage effectively with this group of service recipients. In addition, the African American faith community has been permanently impacted by PRIDE, specifically through the involvement of minority community leaders in the JOBS program, but also more generally through the heightened knowledge of community-wide resources. Lorain's innovative outreach to minority communities and faith-based resources was an asset of this project.

Training in new evidence-based services was possible due to the CRRI funding. There are new service capabilities in the Lorain community as a result of this project. Similar allocations of resources such as funding, advisors, and consultation could be useful in other communities for facilitating and supporting the improvement and transformation of local service delivery systems.

A decentralized model like the one in Lorain created some challenges for the overall evaluation of the PRIDE program and it would not be encouraged again in future projects. The evaluator reported that the software package used for data collection was fine, but because so many agencies were involved, there was too much room for human error. Moreover, there was no easy way to impose order on the process since the reporting structure for the program was similarly decentralized. The attempts to collect standardized data in a timely way were ultimately unsuccessful.

As at the other two sites, PRIDE staff learned that the JOBS program did not carry the same stigma as behavioral health supports; as a result, it proved to be an effective portal for some participants to enter into behavioral health services. We believe this to have been an inadvertent innovation of CRRI, but certainly one that is replicable in other settings.

3.3. Union City Cares: Union City, New Jersey

The Union City CRRI project, variously called "Strengthening Union City's Families" or "Union City Cares," offered strong outreach to Spanish-speaking residents using a model of service delivery embedded in well-known and trusted community settings. CRRI staff were located in the Mayor's office and in existing settings such as the school system, One-Stop Center, and NHCAC's FQHC. During the 28 months of active project operation, the project also included on-site substance use and mental health service programs for high-risk high school-age youth.

Staffing Strategies

The CRRI project was launched at a critical juncture in Union City, which was facing budget cuts to service programs at a time when community needs were increasing due to the Great Recession. Initial project management efforts focused on creative ways to leverage existing resources, such as identifying staff and program space across organizations that served the same target populations. Identifying the right staff to hire for the project was also critical since the project included a diverse array of tasks. The first staff hired had to be proficient in Spanish, since they were to conduct screening and outreach in the community. In addition, the steering committee worked to hire a combination of enthusiastic young staff for some of the community outreach activities and also hire more experienced staff that would provide clinical services. This mix created a strong composition of staff.

Most new staff became employees of the NHCAC with the exception of the school-based substance use counselors who were Hoboken Medical Center employees. As new staff were hired, most of the new CRRI services were embedded in existing community service settings like the One-Stop center, the FQHC at NHCAC, and the Union City School District (UCSD). Placing project staff in these locations created an opportunity for enhancing agency staff awareness about the signs of depression and substance use and increased the number of staff who could refer clients for help. Part of the implementation process was providing time and support for the new staff to get integrated into their settings. Another initial implementation issue was the attention to professional development and to determine adequate ongoing supervision for clinical staff. Hoboken University Medical Center was able to supervise staff in the school system.

Impact of Staff Training

The project staff and other community professionals were trained in a number of new approaches, all of which helped to increase the community agencies' capacity to identify and respond to substance use and behavioral health issues. The staff from several community programs were trained in several SBIRT components, including screening tools, the use of cognitive behavioral interventions, and motivational enhancement therapy. One of the special training elements was a 2-day clinical skills training program on suicide risks. Project staff were also trained to conduct the screening that was central to the outreach portion of the project. The screening tools used were all available as Spanish versions and included:

• Alcohol Use Disorders Identification Test (AUDIT);
• Drug Abuse Screening Test (DAST); and
• Center for Epidemiologic Studies Self-Reported Depression Scale (CES-D).

For prevention efforts, two Strengthening Families programs were developed for parents of youth ages 6-11 and 12-16. Other treatment and interventions resources that were established for the CRRI project included employment resources, substance use, and behavioral health interventions as described in the section on service delivery.

Community Partnerships

The emphasis on collaborative partnerships for the CRRI model encouraged agencies in Union City to work together. Housing part of the program in the Mayor's Office developed close political ties that later served to assist the project. Partnerships were fostered by co-locating staff in several key agencies. Although the initial partnerships were built upon long-standing relationships among a few core agencies, specific issues for this project lead to the establishment of several new community partnerships. Financial support through the CRRI initiative ensured that staff had time to create and maintain these relationships that were central to the program.

Through the duration of the project, new relationships were established between NHCAC and the One-Stop Career center and with the juvenile court. The project also created a referral pathway for families of students in the UCSD program and developed a new communication infrastructure between the schools and substance use providers. Some of the partnerships evolved overtime, and sometimes resulted in the sharing of needed resources. For example, Corrections staff discussed the possibility of providing space in which to operate future programs.

The collaborations addressed special issues, for example, strategies to reach the large Spanish-speaking population, finding ways to overcome stigma about behavioral health and substance use services and promotion of help-seeking behaviors. Designing ways to overcome stigma was an important priority for staff who indicated that among the male Spanish-speaking population it is often difficult to connect consumers to much needed services. Community partnerships also worked together to determine service delivery strategies for how to respond to the high service needs of undocumented individuals. Embedding the new CRRI services within agencies already servicing the Spanish-speaking population was useful for these new outreach efforts.

Outreach and Screening Strategies

During the final Westat site visit, the Union City project team noted the critical impact of its outreach and social marketing efforts. The CRRI tasks during implementation included the development of marketing information, staffing the screening sites, and working with other organizations to conduct outreach and screening. These outreach and screening activities focused on prevention for both mental health and substance use issues and promotion of help-seeking behaviors. During outreach and screening it was very useful to work within general primary care sites. Screening and outreach extended throughout the community and included sites such as WIC and TANF offices, city hall, health clinics, veterans and homeless shelters.

Outreach efforts also included a community-wide distribution of outreach materials. Flyers went to every household in town; bilingual posters were placed at bus stops and businesses; and project staff created brochures and a video. The development of marketing materials was conducted carefully to include Spanish-language versions. Outreach staff identified creative ways for spreading the word about the project at the local level in store front businesses and places where people congregate. Specific attention was given to reaching lower income and undocumented populations by visiting laundromats and conducting outreach in other non-traditional screening settings. Brochures were also shared throughout the PTA in schools. Other media channels, like radio and newspapers, were also used but staff felt the face-to-face outreach was best. Screening at health centers also was noted as an excellent location.

The early focus of this project was on community outreach and prevention of substance use and mental health issues and included a targeted effort on adolescents. As the project evolved, a greater focus was placed on the need to provide employment strategies.

Service Development and Delivery

The networking and collaboration among organizations required as part of the CRRI was noted as a positive effect of participating in the Initiative. CRRI staff noted that coordination is profoundly important among otherwise fragmented services for families in need. Establishing additional resources through the newly hired staff and embedding these new resources in existing agencies helped to support a new sense of potential and encouragement to address sensitive behavioral health issues. By offering bilingual screening and also expanding treatment resources to include bilingual adult mental health and substance use disorder treatment the project also increased the availability of behavioral health and substance use disorder treatment for all city residents.

The focus of the project on employment services began with staff training in an evidence-based job search and support program called JOBS provided through the University of Michigan. This training was designed to teach tools to enhance job search skills, job search motivation and confidence of both unemployed and underemployed residents. Locally, this JOBS program, named HIRED, was delivered in both English and Spanish. Working closely with the One-Stop Career Services was also a key element at North Hudson. Project staff reported that job services and resources were much more needed and beneficial than Strengthening Families.

The program also focused on families. Because of economic challenges, some of the parents had less time to supervise adolescents. With less supervision youth can become involved in high-risk behaviors such as drug and alcohol use. Staff noted that many parents had not finished high school and were determined to avert this problem and wanted resources to help provide prevention and treatment to the children while maintaining them in an educational program.

The project established an adolescent substance use program through the school district and this had a very high impact. Hiring experienced substance use staff allowed new or stronger connections to be built between the schools and the courts, welfare, child behavioral health, and NHCAC. The high school program focused both screening and counseling on site. Establishing this program in the schools facilitated several changes and ultimately expansion in services. Other services for families included offering the Strengthening Families initiative. The original length of this training for families was found to be too long of a commitment for the parents in Union City and so during the project the staff tried modifying the duration of the curriculum. Even with this modification it proved not to be a frequently desired voluntary program. Families experiencing a job loss crisis were most interested in ameliorating that first before considering other supports. More requests and interest was shown for job-seeking resources in Union City and in all three sites.

Innovations

The Union City project found that embedding new services within well-known community agencies already serving many Spanish-speaking clients was a useful innovation for gaining trust with the immigrant community. By placing access to information about behavioral health in more routinely used general agencies like the North Hudson primary care setting, potential CRRI clients were able to receive screening without having to go to a strange, new place.

Using screening to identify more than one service need in new client referrals helped to design a new approach that could help clients cope with long waiting lists for behavioral health services. After screening, a new client could be referred to a variety of available services. This allowed CRRI participants to benefit from other services during a vulnerable time while they were waiting to receive behavioral health supports.

A unique innovation of the Union City project was the provision of substance use services to the youth, on-site in the schools. Establishing new school-based substance use services was seen as a best-practice concept by the project staff and was also identified an excellent recruitment source for troubled families. This approach was tailored to meet the very unique needs of this community. Students known to be using drugs and not in treatment are subject to state mandated suspension. Substance use services in the schools allowed these students to stays in school because there was no waiting period for on-site services. This provided a tool to meet the long-standing challenge of lowering high school dropout rates. It aligned with a strongly held community value of providing support to help youth to complete high school.

Helping clients gain economic stability was empowering and receiving this job-seeking help was noted as less stigmatized than traditional behavioral health services. Union City project staff, similar to the CRRI programs in Lorain and Fall River, found that job-seeking help was more frequently sought because it was much less stigmatized than either behavioral health or substance use assistance.

Political support through the Mayor's Office was useful and was a new portal into services for families. It was important for the Mayor's staff to learn more about the CRRI initiative and it broadened the understanding of this important area and the benefit of outreach and screening as a tool to address stigma and bias as disincentives to help-seeking. However, obtaining adequate support from staff was difficult and one result of the project was that the Mayor's Office decided to establish a new Grant Coordinator so that future grant projects would have easier access to support from the Mayor's staff.

Challenges

Implementing the CRRI effort in Union City confronted several challenges. During the initial implementation there were some delays in filling all the staff positions. Hiring bilingual staff who would meet education and licensing standards was difficult and time consuming. Unfortunately this project did experience some staff turnover, a difficulty for a short-term program. Other difficulties included coping with obtaining supervision for some of the clinical staff and responding to staff turnover. Delays were also experienced in the launch of the media awareness campaign.

Service delivery was initially hampered by a long waiting list to obtain treatment although ultimately the addition of the CRRI grant staff decreased the length of the waiting period from 6 months to 2 weeks. Program challenges also included delay in launching the Strengthening Families. This was partly due to recruiting clients to participate in the lengthy classes. The length of the Strengthening Families parenting programs was not liked and project staff tried to alter how it was presented to reduce the time commitment. Even with these modifications, the program was not well received and staff ended it early. Staff also noted the challenge of an insufficient number of adolescent residential treatment options for behavioral health treatment. Finally, staff noted that meeting the requirements for follow-up interviews for the CRRI project was a challenge. This was difficult because finding some clients who received substance use services was often problematic due to many relocations experienced by the clients.

Sustainability

Several aspects of the CRRI project in Union City will continue. For example, at least six staff of CRRI will remain in new positions and continue to share their expertise in the community. In addition, service agency staff have a newfound appreciation for preventive care and the value of increasing intellectual capital in the community through staff training. CRRI provided a stimulus and an incentive for various agency staff to collaborate with mental health and substance use organizations, the FQHC and the city school district and these connections will continue. During the placement of the substance use program in the schools many other staff, counselors and teachers as well as parents and youth received education about the importance of both prevention and treatment strategies and this level of awareness and knowledge will remain. New linkages with the Department of Corrections were developed during the project and these connections will continue.

The CRRI grant has also leveraged additional resources. Union City applied for and was awarded funding for a Drug Free Communities grant that will help with the sustainability of some elements of the adolescent prevention work.

Finally, the benefit this project revealed of working closely with the Mayor's Office was noted. So strong was the effect of this that a new position was created at the end of the project to foster the involvement of the Mayor's Office and staff in future grant projects.

Many of the tangible services will not be continued due to a lack of funding. The HIRED program will not continue, and the one job expert position at the North Hudson program has now moved to a position with the Veterans Administration. Screening will not be able to continue due to lack of funding, although the community appreciation for the usefulness of screening and for community outreach was engaged. Sustainability of the school program at the same level is doubtful as the substance abuse counselor took a new position and the schools have not filled the position. Substance use and mental health services for adolescents in Union City remain problematic. Programs for youth are limited and courts tend to prefer services in hospitals; these services are limited, however, and are often only provided outside of the county. This creates challenges for families on limited incomes who often have transportation difficulties or work multiple jobs including night shift positions. Transportation, especially for the high school program and after school care, is another critical service gap in this community.

Lessons Learned

Participating in the project allowed all the collaborating agencies and staff to know their community better. The many requirements of the CRRI grant for establishing collaborative partnerships, conducting screenings, or distributing surveys led individuals and organizations to examine their entire community in much greater detail. Examining how the behavioral health needs of the community are met during times of economic distress was profoundly informative and the overall evaluation process was a powerful educational tool.

This collaborative effort was important to address the reluctance and stigma attached to help-seeking for mental health and substance use services among the local Hispanic immigrant, male population. One of the important results of this CRRI effort was how the project emphasized the need to continually outreach and share mental health and substance program resources with the wider community. There is increased community awareness of the how depression, substance use, and unemployment during times of poor economic conditions are connected.

Organizations in Union City now recognize how various resources were profoundly useful to people in need. For example, job-seeking and support services are essential resources along with mental health and substance use services. In addition, the CRRI project provided the resources so that financially strapped organizations could place staff out into the community, which was particularly effective in getting the word out to the residents. Adapting all outreach and treatment resources for Spanish-language community members and hiring fluent bilingual staff to meet the specific needs and values of this community was very useful for the community.

It was unusual to envision a high school substance use program as a tool to recruit more community families into accessing mental health and other services, but through understanding the needs of this community deeply, the usefulness of this approach became a reality. This program found that involving parents was a powerful way to educate the adults about their children's -- and perhaps their own -- behavioral health needs. Program flexibility and honoring community wisdom and leadership for establishing tailored interventions in the future was recommended by the program participants.

Project staff indicated their belief that this project did result in a more "normalization" of help-seeking behaviors. Offering grant opportunities similar to CRRI for other communities would allow them to experience the same learning about how screening, coaching on help-seeking, reducing stigma, and coordinating service systems to be more collaborative can lead to improved client usage and outcomes.

4. Individual and Community Data

This chapter presents findings from Westat's analysis of the full set (from January 2011 through June 2013) of grantees' SAIS data. We begin in Section 4.1 with a discussion of the program evaluation and a description of SAIS data records collected in all three sites. This is followed in Section 4.2 with an examination of the demographic profiles by site of all of the clients who had intake records. Here we highlight some interesting variations across sites, as well as an important finding related to client racial identification in one site. In Section 4.3, we focus solely on the cases for which there were follow-up interviews and our analysis looks at changes in health and employment status for this group. Section 4.4 presents an analysis of program effectiveness. Because of the way the SAIS data are collected, our analysis is restricted to those clients for whom we received a discharge interview record. We first look at the disposition of the discharges by site, reviewing the reasons offered for discharge (e.g., client terminated due to lack of participation, client moved). We then analyze the effectiveness of two program services: employment outcomes for those clients who received job supports and training, and reported changes in substance use and abuse for those clients who participated in substance use interventions. In Section 4.5 we review the community-level data, the approaches used by each of the three sites to gather this information, and the implications of these data for assessing community-level impact.

4.1. Overview of the SAIS Records

Throughout the duration of the CRRI project, each of the three sites collected information from participants using site-specific evidenced-based screening instruments and the SAIS data system, which allows SAMHSA to meet its GPRA reporting requirements. At the time of program enrollment, grantees collected various baseline information on clients, including demographic characteristics, employment status, source(s) of income, and any self-reported behavioral health issues (e.g., mental health symptoms, alcohol abuse, and drug use). Grantees were required to contact clients 6 months post-enrollment (whether the client continued to receive services or not), which resulted in a note of the follow-up being recorded in the client's record. Follow-up interviews, however, were conducted post-discharge and could occur 6 months after enrollment (as in Fall River) or later. Only through these interviews did project staff obtain information about any changes in the individual's employment, physical health, or behavioral health status. Finally, information about services received by clients was only recorded when a staff conducted discharge interviews.

Understanding the distinctions between intake records, follow-up interview records, and discharge records is critical to understanding the various analyses that are presented in this chapter. Intake records provide baseline information on that subset of individuals who were screened in the community and who followed through on the referral to treatment or services. Across all three grantees, there was a total of 1,560 intake records in the SAIS data Westat received from SAMHSA. We culled 22 records from this set: five had no age information (so it was unclear if this was an adult or youth) and 17 from Lorain were clients under the age of 18. Unlike Union City, Lorain's program focused primarily on the city's adult population. Thus, in order not to skew the adult outcomes, we opted to remove these cases from the analysis. As a result, we have baseline data for 1,538 enrolled clients across the sites. Figure 4-1 indicates the number of intakes conducted by each grantee, with adults and youth in Union City represented separately.

FIGURE 4-1 Adult and Youth Intakes Sorted by Site

While the intake records provide baseline information about the clients who enrolled in each of the programs, change in status can only be determined for those clients for whom we have both an intake record and a follow-up interview record. As indicated in Figure 4-2, there are 478 clients for whom no follow-up interview records exist; thus, we can only assess the change in status for 1,060 enrolled clients. Figure 4-2 also indicates the number of follow-up interview records available for each site. There is a noticeable difference in the percentage of enrolled clients lost to follow-up across locations: The highest rates of follow-up are for Fall River, where 41 of 286 clients (14 percent) were lost to follow-up, and among Union City youth, where 45 of 240 enrolled youth (or just under 19 percent) did not have follow-up interviews. Fall River clients received case management services, an intensive service delivery approach that generally resulted in a strong relationship between the staff member and the client. Few clients were lost under this delivery model. Union City youth, by contrast, were essentially a "captive audience" as they were required to participate in services to address substance use if they wanted to stay in school. Although quite a few youth did not have follow-up interviews, overall the retention rate in the program was very good.

FIGURE 4-2 Follow-Up Rates

Outcomes assessments for both Lorain and Union City adults are more challenging, with nearly 40 percent of enrolled clients in each site not participating in a follow-up interview. There are a couple of explanations for this. First, reportedly early on in each program, project staff enrolled clients into the program at the point at which they were screened and referred for services. Many of these individuals simply never followed through on the service referral and were lost to follow-up from the outset. Both programs subsequently altered their policies and formally enrolled clients only after they had actually participated in program services (Lorain discovered that many individuals showed up for one session and did not return, and thus began enrolling clients only after they showed up for a second visit). In addition, it was reported for Union City that the program requirements around completing follow-up interviews were unclear to staff early on, which resulted in a high rate of loss to follow-up. Once the issue was clarified, the follow-up rates for Union City adults improved significantly.

To assess the effectiveness of the services delivered under the CRRI grants, the SAIS dataset needs to contain an intake record (baseline), a follow-up interview (any change in baseline characteristics 6 months later), and a discharge record, which contains the information about the services each client received. As indicated in Figure 4-2, there are 813 clients for whom we have a complete set of records. In Fall River, the follow-up and discharge interviews appear to have occurred at the same 6-month point in time, so all 245 clients who have follow-up records also have discharge records. The other programs do not demonstrate this one-to-one correspondence. Among Union City adults, for example, 76 percent of clients with a follow-up interview also have a discharge record and nearly 82 percent of youth have all three records available for analysis. In Lorain, 53 percent of clients who have a follow-up interview also have a discharge record. There are several explanations for these disparities: First, and given the employment-related concerns of enrollees (or, for youth, their parents), it is quite possible that many individuals simply left the area in search of other work opportunities. Second, some of the individuals were discharged for failing to participate in their selected programs. Many of these clients might have been hard to find 6 months after their enrollment. Third, and for Lorain only, many clients who enrolled in the PRIDE program were eligible for more than one service. Westat was told that clients were not formally discharged from the program until they had availed themselves of all services for which they were eligible. Because there was no time limit for service receipt, clients who were still eligible for services simply were not discharged from the program. In addition, because PRIDE services were decentralized (i.e., the program was run by a board, not a service provider agency), staff reported that it was not always clear among providers which agency was responsible for completing the discharge interview. Thus Lorain's numbers likely under-represent the number of clients who benefited from PRIDE services; however, Westat is only able to conduct an assessment of their programs' effectiveness for those clients who had the full set of records.

4.2. Analysis of the Intake Records

Timeframe of Intakes by Site

The CRRI contract called for sites to begin enrolling clients no later than 4 months after the grant award date (September 2010). Nevertheless, sites varied slightly as to when they began implementing their programs. The graphs in Figure 4-3 indicate the "flow" of the intakes in each site over the length of the project. Fall River began enrolling clients on January 24, 2011. Lorain began a month later, on February 26, 2011.

For both of these sites, the figures indicate that a surge in enrollments began a couple of months later. In Union City, the program enrolled its first young person on February 24, 2011, followed a week later by its first adult (March 2, 2011). Data suggest that once Union City began full implementation of its program, it began enrolling a large number of youth and adult clients.

The reader also will note that follow-up activities (noted by the red lines in the graph) are seen typically 6 months after each of the intakes surges. However, as noted in Section 4.1, the ability to conduct follow-up interviews early in the program varied across the sites. Fall River, for example, used a case management approach that allowed for a nearly seamless follow-up process. Lorain's first follow-up peak occurred 6 months after its program began, although had a lower amplitude than its preceding intake pattern. Likely this drop-off can be accounted for by the premature enrollment of clients, which resulted in a high loss to follow-up rate. Lorain changed its enrollment policy in response, and the graph clearly illustrates how successful this new process was. Finally, in Union City, there is a noticeable lag in follow-up activities early in the program's implementation. As noted, staff also reported enrolling clients prematurely and were not certain about the need to conduct follow-up interviews. Once this grant requirement was clarified, staff reportedly tracked down many of these early clients and conducted follow-up interviews. This perhaps accounts for the significant increase in follow-ups in the spring and early summer of 2012.

FIGURE 4-3 Site Intake and Follow-Up Records by Month

Union City's program ended earlier than the other two sites, thus the latest intake records Westat received for this grantee were dated March 15, 2013 and April 24, 2013 for youth and adults, respectively. For Fall River, the latest intake record was dated June 14, 2013 and for Lorain the last record was received on June 26, 2013. Follow-up interviews trailed off accordingly.

Gender Differences by Site

The proportion of males to females at intake varied considerably across sites as shown in Figure 4-4. In both Fall River and among Union City adults, more women than men were represented at intake. In Fall River, however, only about 15 percent more women than men enrolled in the program. Given the large Portuguese population in the community and the reported tendency for Portuguese men to be reluctant to seek assistance, the relatively even gender representation within Project ASSIST is a testament to the successful outreach and engagement strategies of program staff. In Lorain and among Union City youth, there are significantly more males than females at intake, while among the Union City adults women are disproportionately represented in the data. During site visit interviews, staff commented that their outreach effort was more easily conducted with the female Hispanic population in Union City. Strong stigma and reluctance to seek help were noted by grantee staff as barriers to reaching the male Hispanic population in that city. Although the youth enrolled in the program were primarily Hispanic, the greater representation of males in the school-based program likely was because the services were mandated.

FIGURE 4-4 Gender Representation by Site, at Intake

Ethnic/Racial Differences by Site

Figure 4-5 shows the variation in enrolled clients' self-reported race and ethnicity across the three communities.⁸ Fall River's records indicate that the vast majority of individuals who received an intake record consider themselves non-Hispanic and White. The majority of Union City's adult and youth populations consider themselves Hispanic. Lorain's clients evince the greatest diversity among the three communities, with Whites, Blacks, and Hispanics represented in the intake records. Notably, as discussed in the site summary in Chapter 2, there were differing opinions among project leadership about where the program should focus its efforts. While one believed the program should focus on the newly unemployed citizens of the community, the project director believed that long-term unemployed residents also were being adversely affected by the Great Recession. As a result, she reportedly concentrated her efforts on Lorain's African American community, which had been hard-hit by years of economic downturns. The data support this statement, as 32 percent of PRIDE clients self-identified as African American compared to 17.6 percent in the population according to Census 2010.⁹ Hispanics are represented among clients at approximately the same rate as one might expect based on Census data (approximately 25 percent), while Whites are significantly under-represented in PRIDE (33 percent among PRIDE clients compared to 67 percent in the Lorain City population overall).

FIGURE 4-5 Ethnic/Racial Breakdown by Site, at Intake

** Other = All other races (Asian, American Indian, Hawaiian Native, etc.).

Age Differences at Intake by Site

The average age of adult clients was consistent across the three grantees: 38.2 years in Fall River, 38.3 years in Lorain, and 39.9 in Union City. The average age of Union City youth was 14.7 years old. If youth had not been separated from the Union City adults, however, the average age for all program participants in that city would drop to 32.3 years. By analyzing the youth data separately, we see that all three sites were, on average, serving the same age cohort in their communities.

Substance Use Issues at Intake by Site

Self-reported use of illegal drugs and/or alcohol highlights differences among the populations within each city, as can be seen in Figure 4-6.¹⁰ On the whole, Union City adults reported lower levels of drinking to intoxication or use of illegal drugs than enrolled clients at the other two sites. Union City youth, however, and by virtue of the program in which they were enrolled, reported much higher rates of substance use than the other three populations.

FIGURE 4-6 Self-Reported Use of Illegal Drugs and Alcohol Use to Intoxication at Intake, by Site

Employment Status at Intake

Table 4-1 depicts enrolled clients' employment statuses at intake for each site. In all three sites, there is clearly a large proportion of the client population that is unemployed and looking for work. This finding is consistent with the awarding of the grants to three communities that were experiencing significant economic distress subsequent to the recession. Interestingly, however, Union City evinces a much larger proportion of its client population that is employed either full-time or part-time. It is Westat's understanding that in this grantee site, many adults were enrolled in the Strengthening Families program because of their children's identified behavioral health needs. That is, school-age youth caught using alcohol or drugs were recommended for the Strengthening Families program and their parents came as a result of the child's referral. In fact, during the Year 3 site visit, Westat staff learned that many of the enrolled youth were "out of sight" of their parents and thus engaging in problematic activities because the adults were working more than one job to make ends meet. The high proportion of employed adults in Union City is thus likely a reflection of this aspect of adult enrollment into services.

TABLE 4-1 Clients' Employment Status at Intake

	Fall River   (n=285)	Lorain   (n=454)	Union City Adults   (n=554)
Employed full-time	3%	9%	20%
Employed part-time	9%	6%	18%
Unemployed, looking for work	78%	67%	46%
Unemployed, disabled	6%	7%	4%
Unemployed, volunteer work	1%	1%	1%
Unemployed, retired	0%	1%	1%
Unemployed, not looking for work	2%	6%	10%
Other	0%	3%	1%

 

4.3. Outcomes Analysis

To begin exploring the impact of grantees' efforts, Westat looked first at those intakes for which there existed a follow-up record. Of the 1,538 records gathered at intake we have a total of 1,060 follow-up records in our dataset. The number and percentage of follow-up cases per site is highlighted in Table 4-2. Note that only for Fall River and Union City youth did the grantees meet SAMHSA's requirement for an 80 percent follow-up rate. Across all grantees' programs, the follow-up rate is 69 percent.

TABLE 4-2 Number of Intakes and Follow-Up Interviews, by Site

	Fall River	Lorain	Union City   Adults	Union City   Youth	Total
Number of intakes	286	458	554	240	1,538
Number of follow-up interviews	245	278	342	195	1,060
Percent of cases with follow-up records	86%	61%	62%	81%	69%

 

Alcohol Use

The first SAIS question for alcohol use reads as follows: "During the past 30 days, how many days have you used any alcohol." Answers could potentially be very misleading for the adult client population. For example, a client might answer "4 days," but have engaged in binge drinking on each of those occasions; alternately, a client might answer "10 days" but had only one drink each of those days. This question does not allow for distinctions to be made between problematic and non-problematic use among adults. Alcohol use among youth, however, is illegal and thus clearly is problematic. Among the 195 Union City youth for whom both intake and follow-up records were available, 62 (32 percent) reported any alcohol use in the 30 days prior to the intake interview. At follow-up, however, only seven (4 percent) of the 195 youth reported consuming alcohol. Although one must always be mindful of the significant potential among this population for providing a socially desirable survey response, these numbers nonetheless are very encouraging.

Use of Illegal Substances

The drug use question was in the same format as the question about alcohol use, but included a list of drugs to serve as "fill" items for the question. Thus, it read, "During the past 30 days, how many days have you use any of the following?" which included marijuana, crack/cocaine, heroin, as well as a variety of prescription pain killers (e.g., Percocet, Tylenol 3). With the exception of the youth, these numbers are proportionately quite small. For example, in Fall River, of the 245 clients for whom ASSIST had both an intake and follow-up interview, only 29 clients (less than 12 percent) at intake reported having used an illegal substance in the previous 30 days. In Lorain, 53 of 278 (19 percent) reported illegal substance use at intake, and among Union City adults only 7 percent so indicated. Appropriately (given the program focus), about half of the Union City youth reported having engaged in illegal drug use in the 30 days prior to the intake interview.

Notably, these numbers are significantly different at 6-month follow-up, as shown in Figure 4-7. Once again, although the potential remains for clients to provide socially desirable responses when asked about drug use, these numbers are very impressive as the client populations' substance use behaviors decreased over time.

FIGURE 4-7 Percent of clients Reporting Substance Abuse at Intake and Follow-Up

Stress, Reduction in Activity, or Emotional Problems As a Result of Alcohol or Drug Use

For those clients who indicated they had used alcohol or illegal drugs over the past 30 days, the SAIS tool also asks about the extent to which the respondents' alcohol or drug use has adversely affected their lives. For example, respondents are asked if over the past 30 days their substance use has caused them stress. Figure 4-8 shows the number of clients in each site who reported their substance use over the previous 30 days had resulted in increased stress. Figure 4-9 indicates population-level improvement at follow-up. Of particular importance are those individuals who reported at follow-up that they had not used either alcohol or drugs in the past 30 days.

FIGURE 4-8 Individuals Reporting Stress Because of Drug or Alcohol Use in the Last 30 Days at Intake

FIGURE 4-9 Individuals Reporting Stress Because of Drug or Alcohol Use in the Last 30 Days at Follow-Up

 

Clients who reported using alcohol or drugs in the past 30 days were also asked the extent to which their substance use had affected either their activity levels or caused them emotional problems. The change between intake and follow-up on each of these measures can be seen in Figure 4-10, Figure 4-11, Figure 4-12 and Figure 4-13, respectively.

FIGURE 4-10 Individuals Reporting Reducing or Giving Up Important Activities Because of Drug or Alcohol Use in the Last 30 Days at Intake

FIGURE 4-11 Individuals Reporting Reducing or Giving Up Important Activities Because of Drug or Alcohol Use in the Last 30 Days at Follow-Up

FIGURE 4-12 Individuals Reporting Emotional Problems Because of Drug or Alcohol Use in the Last 30 Days at Intake

FIGURE 4-13 Individuals Reporting Emotional Problems Because of Drug or Alcohol Use in the Last 30 Days at Follow-Up 

 

Figure 4-14 and Figure 4-15 present a composite across all three measures at intake and follow-up, respectively.

FIGURE 4-14 Individuals Reporting Any Stress, Reduction in Important Activities, or Emotional Problems Because of Drug or Alcohol Use in the Last 30 Days at Intake 

FIGURE 4-15 Individuals Reporting Any Stress, Reduction in Important Activities, or Emotional Problems Because of Drug or Alcohol Use in the Last 30 Days at Follow-Up 

Self-Reported Mental Health Symptoms

All individuals were asked during the intake interview to report on their mental health status. The SAIS question reads: "In the past 30 days, not due to your use of alcohol or drugs, how many days have you experienced the following symptoms?" Symptoms included depression, anxiety, hallucinations, trouble remembering or concentrating, and attempted suicide. Overall, most of the indicators were rarely reported as having been experienced by clients at enrollment, with four exceptions: 35 percent of the cases across all three sites (including youth) said at intake they had experienced cognitive difficulties in the previous 30 days, while just under 10 percent of clients said at intake that they had had difficulty controlling their violent behavior. These numbers drop to 21 percent and 5 percent, respectively, by follow-up. Among the other symptoms, the highest rates of endorsement were for depression and anxiety. Figure 4-16 and Figure 4-17 show the percentage drop in each of these symptoms between intake and 6-month follow-up.

FIGURE 4-16 Percent of Clients Reporting Symptoms of Depression by Site at Intake and Follow-Up

FIGURE 4-17 Percent of Clients Reporting Symptoms of Anxiety by Site at Intake and Follow-Up

 

Across all four populations, we see a decrease in the percentage of individuals who reported having experienced either of these two mental health symptoms in the previous 30 days. While the percentage of clients reporting depression symptoms drops by about half among Fall River clients and Union City youth, the decline is less among Lorain clients and Union City adults. At present we do not have an explanation for these differences. In each site, the percentage of clients reporting symptoms of anxiety also decreases between intake and follow-up, although in none of the four sites is the decline as dramatic as for depression. It does appear, however, that program participation was correlated with positive behavioral outcomes in the client population as a whole.

4.4. Program Effectiveness Analysis

As noted earlier, complete SAIS cases require an intake record, follow-up interview, and a discharge record. The number of discharge records for each site is depicted in Figure 4-18. When creating discharge SAIS records, staff must assign each client a specific discharge status: the primary status marker is "Completion/Graduate" versus "Termination." Records for all clients who are "terminated" must contain a reason for that disposition status. Table 4-3 indicates the statuses of terminated clients for each grantee.

FIGURE 4-18 Number of Discharge Records, by Site

 

TABLE 4-3 Clients' Status at Discharge, by Site

Analysis of Complete Cases

As indicated previously, a SAIS discharge record for a client is the only source of information within the dataset that describes which services clients received from the CRRI program and at what frequency. Similarly, the follow-up record is the only gauge of an individual's improvement over time. To determine potential links between a client's attendance at specific programs and his/her health and employment outcomes, it is necessary that full sets of Intake/Follow-up/Discharge records for clients (i.e., "complete cases") be available. Table 4-4 shows the number of complete cases for each site and for those participating in employment services or services to address substance use.

TABLE 4-4 Number of Complete Cases by Site and by Service

	Fall River	Lorain	Union City   Adults	Union City   Youth
Total number of completes	245	148	260	160
Receiving employment services	235	92	124	--
Receiving substance abuse services	21	1	58	131

 

Employment Services and Outcomes

This analysis focuses only on those complete cases where the discharge record showed the client received employment or training services. The number of complete cases for employment assessment by site is depicted in the second row of Table 4-4. Note that we have not included any analysis of the Union City youth since the primary focus of these young people is on finishing school, not entering the workforce.

Figure 4-19 depicts the clients' change in enrollment status in school or a job training program between intake and follow-up. Neither Lorain nor Union City indicated any marked increase in clients who were enrolled in such a program. In Fall River, however, the number of clients who were enrolled in school or a training program either part-time or full-time more than doubled during the 6 months that the clients were enrolled in the ASSIST program.

FIGURE 4-19 Percent of Clients Receiving Employment Training Who Were Enrolled in School or Job Training Program Full-Time or Part-Time

Figure 4-20 depicts clients' employment status at intake and follow-up for those individuals who received job training services through the programs. Across all three grantees, employment status showed a very favorable increase between the two points in time. While it is possible that the CRRI employment services made a strong contribution to these increases, we remind the reader that there has been significant recovery in the economy since the Great Recession, including a decline in the unemployment rate. Nevertheless, these gains are impressive and suggest that the CRRI clients were able to take advantage of employment opportunities as they became more available.

FIGURE 4-20 Percent of Clients Receiving Employment Training Reporting Full-Time or Part-Time Employment

Substance Abuse Services and Outcomes

Finally, we examined any changes in self-reported substance use for those individuals whose records clearly indicated that they received substance use disorder services from the grantee. Here we focus only on Fall River and Union City youth (see Table 4-5). For Lorain, the data indicated only one discharged client who had received substance use disorder services. The numbers for Fall River are also small, so they should be interpreted with caution. For example, the number of clients who reported drinking five or more alcoholic beverages in one sitting and who received substance use disorder services increased by one between intake and follow-up. Can we conclude that of the 21 clients at that site who participated in substance use disorder services, one actually began binge drinking as a result of participating in services? It is unlikely, but this "finding" suggests the reader should interpret any findings suggesting causality with caution.

TABLE 4-5 Substance Abuse Services and Related Outcomes

In Union City, there are more complete cases for both adults and youth who received substance use disorder services. For both populations we see a large number of clients who reported using alcohol in the previous 30 days at intake, with a sharp drop in the number who continuing to use alcohol at follow-up. At follow-up, there is a decline in the number of adults who reported binge drinking and the number of youth who reported binge drinking drops to 0 in both categories. These data suggest a positive impact of program participation on adult and youth alcohol use, but may also be the result of participants responding the way that they thought staff would want particularly in how youth answered the follow-up questions. Similarly we see a significant decline in both populations of the number who reported using illegal substances at follow-up. Again, this is an encouraging finding, although it should be interpreted cautiously.

4.5. Community-Level Data

Finally, we turn to the community-level data reported by each grantee. The reader will recall that within the CRRI contract, grantees were required to collect indicators related to any changes in health and well-being in their communities. Some of these indicators were to be obtained from service providers (e.g., police would be asked for the number of domestic abuse calls each year, grantees would request data on the number of drug-related or alcohol-related admissions to area hospitals). In addition, to assess the effectiveness of their media campaigns and outreach and screening efforts, each grantee was to conduct a community-level survey each year of the grant. The data submitted were not comparable across the three sites, thus we look at each grantee's indicators separately.

Union City

Union City ensured that a youth survey and a community-level survey were conducted each of the first 2 years of the grant. Youth surveys were conducted in the High School and Freshman Academy (9th grade) during physical education class 1 day each year in 2011 and 2012. Response rates were reportedly greater than 50 percent each year, thus these data are believed to be representative of the school-age population in this community. Table 4-6 displays the percentage of respondents who answered "Yes" to each question for 2011 and for 2012.

In their final report, Union City noted that separate databases were used for each year and some of the questions varied between the two surveys. They reported that "we are unsure if the differences between the 2 years are statistically significant, but it appears that there was an increase in employment and a decrease in depression, anxiety, and alcohol use among adolescents and their parents in Union City in the year between the survey administrations." It was also noted that some of the changes may be the result of variation in the survey (the questions that were added between 2011 and 2012) and how it was administered, but it is possible that the changes were also the result of the CRRI project. The differences could also be the result of regression to the mean. In any event, we encourage the reader to review these data with cautious optimism.

TABLE 4-6 Union City Youth Survey: Percentage of Respondents Who Answered "Yes" to Each Item in 2011 and 2012

 

Question	2011	2012
Number of responses	1,162	1,092
Parent/guardian is unemployed and looking for work	19.5%	13.6%
Parents worry a lot about work and money issues	57.3%	55.9%
In the last 4 weeks, I've been sad a lot	28.5%	21.1%
In the last 4 weeks, I've had trouble sleeping	29.4%	27.0%
When I have problems, there are people in my family who help	72.5%	74.6%
When I have problems, there are people outside my family who help	76.8%	79.1%
In the last 4 weeks, I've felt nervous a lot	22.3%	20.0%
In the last 4 weeks, I got drunk	22.7%	18.6%
In the last 4 weeks, I used marijuana at least once	13.8%	14.9%
Parents often get high or drunk	9.3%	6.2%
Parents are often sad or stressed	33.0%	30.5%
If someone in my family needs help for depression, anxiety, alcohol or drugs, I know where to get help	40.8%	42.0%
Someone in my family has gone to NA, AA, Al-anon or Alateen meetings	6.1%	4.9%

In addition to the youth survey, Union City conducted a community-wide adult survey in both 2011 and 2012. Surveys were mailed to 1,500 adults randomly selected from current voter registration lists. The same number of surveys were also distributed in NHCAC's health clinic by randomly selecting undocumented Union City patients who were not eligible to vote. The response rate in both years was very low, at about 11 percent, thus making the results difficult to interpret. Nevertheless, the results are almost opposite of what the youth reported, as indicated in Table 4-7, below (again the reader should note that the percentages displayed for each item reflect the "Yes" responses to each survey item). With the exception of the first question on employment, the data suggest adults were in greater emotional distress in 2012 than in 2011. We encourage the reader to be very cautious when attempting to draw conclusions about these data. CRRI was one aspect of community dynamics during this time period, but there were numerous other contextual factors that would need to be taken into account to understand what was influencing community well-being.

TABLE 4-7 Union City Adult Survey: Percentage of Respondents Who Answered "Yes" to Each Item in 2011 and 2012 

Question	2011	2012
Number of responses	568	351
I am unemployed or looking for more work hours	58.2%	54.4%
My worries about work and money have grown in the last year	70.1%	70.5%
In the last 4 weeks, I've been sad more than usual	39.2%	49.7%
In the last 4 weeks, I've had trouble sleeping	44.6%	51.7%
When I have problems, there are people in my family who help	69.9%	65.2%
When I have problems, there are people outside my family who help	48.7%	48.7%
In the last 4 weeks, I've felt nervous a lot	32.7%	42.3%
In the last 4 weeks, I've been drinking more than usual	7.8%	9.7%
In the last 4 weeks, I got drunk at least once	8.3%	15.4%
In the last 4 weeks, I used marijuana at least once	3.4%	7.7%
In the last 4 weeks, I used cocaine or heroin at least once	1.6%	5.7%
In the last 4 weeks, I used pain medication for reasons other than pain	14.8%	23.9%
Someone in my family has gone to NA, AA, Al-anon or Alateen meetings	7.0%	13.3%
If someone in my family needs help for depression, anxiety, alcohol or drugs, I know where to get help	37.3%	41.0%
I know that being unemployed may cause people to drink more, use drugs or become sad or nervous	67.6%	67.0%

Finally, Union City did report several community-level variables, as displayed in Table 4-8, although there is longitudinal data only for the first and last variables. Moreover, these data are inconclusive; domestic violence does not show a clear trend across the 4 years, and indeed begins to spike again in 2012. Reports of child maltreatment generally trend downward, although there is a slight increase between 2011 and 2012. No conclusions can be drawn from these indicators about the community impact of the CRRI grant in Union City.

TABLE 4-8 Community Variables for Union City 2009-2012

	2009	2010	2011	2012
Number of domestic violence incidents	544	556	477	522
Number of alcohol-related hospitalizations	*	1,010	*	*
Number of reports of child abuse and maltreatment	188	189	159	162

 

Lorain

Lorain's PRIDE provided several community-level indicators in their final project report, as indicated in Table 4-9. Several of these indicators came from their community surveys, which were conducted with 300 households in both 2011 and 2012.

TABLE 4-9 Community Variables for Lorain 2010-2012

As with the Union City data, these numbers are inconclusive; domestic violence incidents and calls to the crisis hotline regarding suicidal individuals both decline over the time period of the grant, but the number of reports of child abuse and maltreatment increases sharply and respondents' perceived sense of connectedness to the community declines. Westat would draw no conclusions from these limited data and we similarly caution the reader from drawing any connections between the CRRI program and these results. There are numerous contextual and historical factors that would need to be taken into account, and over a longer period of time, to assess any potential impact of the CRRI initiative in this community.

Fall River

Project ASSIST did not conduct a community-level survey. The project also provided the evaluators with some very limited community-level indicators. For example, this site reported a 10.4 percent rate of domestic violence in 2010, but indicated even up through 2012 that subsequent data were "not yet available." Similarly, they reported that the rate of alcohol-related or drug-related hospitalizations stood at 29.5 per 1,000 population in 2008, but failed to report additional data, indicting updated information was "not yet available." Finally, for 2009-2010 they reported a rate of substantiated reports of child maltreatment of 44.9 percent, including 93 percent charged with neglect. Data from subsequent years was "Not yet available." Because Fall River did not report even two points in time, no analysis can be conducted with these data.

Summary

The assessment of potential community-level impacts of the CRRI grants was always dependent on data elements to be collected by the grantees. Union City provided youth and adult survey data, as well as community data elements, across several points in time. Lorain needed some additional support from Westat to get its community survey launched, but eventually did so and was able to report both survey and some limited community data elements. Fall River did not report any community-level data.

Other place-based initiatives have noted the importance of conducting community surveys to assess the impact of the initiative. The Annie E. Casey Foundation, for example, funded a 10-year neighborhood improvement initiative called Making Connections that aimed to enhance various community protective factors (e.g., parental economic stability, community ties) for vulnerable youth. In a post hoc assessment of the initiative,¹¹ the survey was explained as follows:

Making Connections presented several reasons to rely on surveys as a primary evaluation tool: the Foundation hoped to measure changes in civic participation, social ties, and other key outcomes that aren't reflected in administrative records. Moreover, local administrative data weren't defined and collected in the same ways in all Making Connections sites, making it hard to pool or compare data.

To ensure consistency across the grantee sites (i.e., comparable frames and sampling methodology and data collection procedures), the Foundation hired a survey research company to conduct this piece of the evaluation.

Although other recent literature reviewing place-based initiatives indicates that measuring the effectiveness of these efforts remains problematic,¹² there is continued work to develop tools for measuring resilience. These include the Conjoint Community Resilience Assessment Measure, authored by Aharnonson-Daniel and Lahad 2012,¹³ which consists of 32 items on a five point Likert scale that measures several key factors, such as residents' faith in community leaders, a sense of community efficacy ("I believe that my community has the ability to overcome crisis"), the respondent's sense that the community is prepared to deal with emergency situations, the respondent's attachment to the community ("I feel that I belong to the place where I live"), and two items on the quality of relationships between community members.

The Prevention Institute has developed a Toolkit for Health and Resilience in Vulnerable Environments,¹⁴ an assessment tool that measures certain dimensions of community resilience, including residents' mental health, substance abuse, and other aspects of emotional well-being. Communities Advancing Resilience Toolkit (CART), discussed in Pfefferbaum et al. 2013,¹⁵ is both an intervention and a measurement that contains 21 items rated on a Likert scale. Examples include "People in my community are committed to the well-being of the community" and "My community looks at its successes and failures so it can learn from the past." These and other similar tools might be considered by SAMHSA for incorporating into the evaluation of any future place-based initiatives.

Collecting meaningful community-level indicators by using existing datasets has also been pointed out as particularly challenging. For example, in a "best-practices" manual published subsequent to the evaluation of First 5 LA's Best Start initiative, the authors noted the difficulty of identifying public datasets that provide the same information (measured in the same way) across multiple sites with sufficient geographic precision. In addition, and perhaps more importantly, they point out that place-based initiatives are focused on low-income neighborhoods, but cities are comprised of both low-income and high-income residential areas. Even city-specific data may not reveal initiative-induced changes because local-level changes may not prove statistically significant when viewed within the context of a larger, mixed-income population.¹⁶

To address this problem, the Urban Institute has established a National Neighborhood Indicators Partnership (http://neighborhoodindicators.org/) in which they work with local organizations to develop locally relevant datasets. With funding from John D. and Catherine T. MacArthur Foundation and the McKnight Foundation, they (like the evaluators in Making Connections) are working to create shared indicators with common measures that will be collected by all of the locales participating in an initiative. Another source of data that may be helpful for future place-based projects is the National Neighborhood Indicators Partnership to determine if existing (or even future) grantees might be able to capitalize on this work. Examination of this resource during the program design stage may enable the identification of existing standard data across communities.

Finally, the CRRI grants were unexpectedly short-lived. The literature on place-based initiatives (e.g., Making Connections) clearly indicates that they require substantial investments of time to effect any community-wide change. This is because community initiatives are complex and are part of a social change process that is larger than a single agency or even a coalition of agencies. In the CRRI projects, the shortened timeframe for planning, launching, and implementing the programs created a split focus for the sites. On top of that considerable effort, they did not receive expected continuation funding and had to begin planning for sustainability just as they were getting established. The planning committees at each site thus had to figure out how to salvage what had been developed and rework their plans because of the unanticipated ending. Future place-based initiatives should account for the timeframe needed to implement community-wide programs. Grantees need an opportunity to work out an early implementation challenges, including staffing, determining if selected interventions need to be reconsidered (e.g., the demise of the Strengthening Families efforts in all three sites), learning GPRA reporting requirements, and the like. Only once these challenges have been addressed can they put their full efforts into project implementation. And it is only once these programs have been fully established that one can reasonably assess their impacts on the local communities.

5. Summary and Implications

The CRRI was an important and ambitious undertaking. It aimed to help the three grantee communities recover from the behavioral health effects of the Great Recession, as well as arm these cities and their citizens with the tools to better handle any subsequent economic downtowns. Because of the shortened timeframe of the initiative as well as significant challenges with the community data for each grantee, our evaluation is inconclusive regarding the impact of these grants on the broader communities. Findings do indicate, however, that many of the individuals who received services through the initiatives reported improved outcomes in both employment and self-reported behavioral health symptoms. In addition, the grants had a positive impact on the service delivery infrastructure in each of these communities. For example, the many requirements of the CRRI grant for partnerships, screening, and collaborations with new entities led individuals and organizations to examine their entire community in much greater detail. In gaining an understanding about the link between economic distress and behavioral health challenges, new linkages have been formed between behavioral health providers and employment services in each community. Likely these cross-referrals will continue going forward.

Another important aspect of this effort was the project's emphasis on the need to continually outreach and share mental health and substance program resources with the wider community. Through grantees' outreach and screening efforts, there is increased awareness among many residents about how depression, substance use, and unemployment during times of poor economic conditions are connected. In addition, staff in all three sites believed that their efforts had reduced the stigma around help-seeking for mental health services and reached people who would not normally access behavioral health supports. Because outreach and screening are not reimbursable services for providers, however, communities will have to identify financial resources that will allow them to engage in these activities. Offering grant opportunities similar to CRRI to other communities would allow them to experience the same learning about how screening, coaching on help-seeking, reducing stigma and coordinating service systems to be more collaborative can lead to improved service access and client outcomes.

The CRRI grants also provided resources that allowed financially strapped organizations to hire needed staff. In Lorain, for example, several public health nurses were able to return to duty after a layoff and provided critical outreach services to city residents. In Union City, staff adapted all outreach and treatment resources for Spanish-language community members and hired bilingual staff to meet the specific needs and values of this community. Service providers are generally aware of service gaps in their communities, but do not always have the resources to meet their residents' needs. The CRRI projects allowed three communities to take positive steps towards addressing their varied gaps in service outreach and delivery.

Grant-funded projects must always strike a balance between requiring all grantees to engage in the same activities and allowing grantees to modify their approaches to meet the unique needs in their communities. While the variations are problematic from an evaluation standpoint, the flexibility of the CRRI grants resulted in some important innovations. For example, in Union City, the grantee recognized the centrality of youth within the ethos of that city's residents. Rather than focus on serving adults directly, they developed a high school substance use program that also acted as a tool to recruit more community families into mental health and other services. Lorain's project director believed that the African American residents of that community had not only been adversely affected by the Great Recession, but also had long-standing unemployment challenges that had not been adequately addressed by the service delivery system. She thus paid particular attention to this segment of the city's population, despite the fact that this population's employment-related issues were more "chronic" than emergent. Finally, Fall River adopted a case management model to provide "brief" interventions to enrolled clients. Although case management is a much more intensive service delivery model than was adopted by the other two grantees, these 6-month relationships were linked to positive client outcomes and contributed to the collection of a fairly complete set of data for analysis.

The CRRI grants allowed innovative treatment approaches, new community partnerships, and the awareness of a critical connection between economic distress and behavioral health issues among service providers. It is hoped that these accomplishments will contribute to the resilience of these three communities should they face widespread economic challenges in the future.

Notes

1. In order to draw this inference, there would have had to be a randomized control group in each community, or a stronger quazi-experimental design (i.e., a group of individuals whose demographics and baseline characteristics paralleled those of CRRI participants, but who did not participate in any of the CRRI-funded programs). Only a comparison of the outcomes of CRRI clients with non-CRRI clients would have allowed us to make more definitive claims that the programs themselves were responsible for individuals' improvements.

2. A brief summary of the Great Recession can be found on the website of the Economic Policy Institute at: http://stateofworkingamerica.org/great-recession/.

3. See http://www.bls.gov/spotlight/2012/recession/pdf/recession_bls_spotlight.pdf.

4. See http://stateofworkingamerica.org/economic-indicators/national-jobs/.

5. See http://www.city-data.com/business/econ-Fall-River-Massachusetts.html#top.

6. Unemployment data obtained from Lorain's grant application, page 4.

7. See http://www.city-data.com/city/Lorain-Ohio.html.

8. We note that according to the classifications used by the United States Census, race and ethnicity are two different categories (i.e., a person can be Hispanic [ethnicity] and also White [race]). Therefore, the sum for any site in the figure may exceed the number of intake records for that site.

9. See http://quickfacts.census.gov/qfd/states/39/3944856.html.

10. It should be noted by the reader that these are not unduplicated counts. That is, an individual could report drinking to intoxication on the weekends and using illegal drugs during the week with at least one instance in which he both drank to intoxication and used illegal drugs on the same day. Thus the percentages in each column for a site should not be totaled to indicate the proportion of clients acknowledging all substance use or abuse at intake.

11. L. Feister (2011), "Measuring Change While Changing Measures: Learning in, and from, the Evaluation of Making Connections." Annie E. Casey Foundation.

12. See, inter alia, N. Cytron (2010), "Improving the Outcomes of Place-Based Initiatives. Community Investments," Spring 2010, Volume 22, Number 1. Federal Reserve Bank of San Francisco. L. Schorr and F. Farrow (2011), "Expanding the Evidence Universe: Doing Better by Knowing More." Center for the Study of Social Policy. L. Feister (2011), "Measuring Change While Changing Measures: Learning in, and from, the Evaluation of Making Connections." Annie E. Casey Foundation.

13. L. Aharnonson-Daniel and M. Lahad (2012). Interactive Session presenting a novel methodology to assess community resilience developed by The Conjoint Community Resiliency Assessment Collaboration, a multidisciplinary group of senior researchers from various institutions. In Presented at the Second Israeli International Conference on Health Care Systems Preparedness and Response to Emergency and Disasters, TelAviv, Israel, January 2012.

14. See http://www.preventioninstitute.org/component/jlibrary/article/id-96/127.html.

15. R.L. Pfefferbaum et al. (2013), "The Communities Advancing Resilience Toolkit (CART): An Intervention to Build Community Resilience to Disasters." J Public Health Management Practice. 2013 May-June; 9(3): 250-8.

16. "Best Practices in Place-Based Initiatives: Implications for Implementation and Evaluation of Best Start." Juarez and Associations and Harder+Company, Prepared for First 5 LA, November 2011.