Community Resilience and Recovery Initiative: Final Evaluation Report. 4.1. Overview of the SAIS Records

03/27/2014

Throughout the duration of the CRRI project, each of the three sites collected information from participants using site-specific evidenced-based screening instruments and the SAIS data system, which allows SAMHSA to meet its GPRA reporting requirements. At the time of program enrollment, grantees collected various baseline information on clients, including demographic characteristics, employment status, source(s) of income, and any self-reported behavioral health issues (e.g., mental health symptoms, alcohol abuse, and drug use). Grantees were required to contact clients 6 months post-enrollment (whether the client continued to receive services or not), which resulted in a note of the follow-up being recorded in the client's record. Follow-up interviews, however, were conducted post-discharge and could occur 6 months after enrollment (as in Fall River) or later. Only through these interviews did project staff obtain information about any changes in the individual's employment, physical health, or behavioral health status. Finally, information about services received by clients was only recorded when a staff conducted discharge interviews.

Understanding the distinctions between intake records, follow-up interview records, and discharge records is critical to understanding the various analyses that are presented in this chapter. Intake records provide baseline information on that subset of individuals who were screened in the community and who followed through on the referral to treatment or services. Across all three grantees, there was a total of 1,560 intake records in the SAIS data Westat received from SAMHSA. We culled 22 records from this set: five had no age information (so it was unclear if this was an adult or youth) and 17 from Lorain were clients under the age of 18. Unlike Union City, Lorain's program focused primarily on the city's adult population. Thus, in order not to skew the adult outcomes, we opted to remove these cases from the analysis. As a result, we have baseline data for 1,538 enrolled clients across the sites. Figure 4-1 indicates the number of intakes conducted by each grantee, with adults and youth in Union City represented separately.

FIGURE 4-1 Adult and Youth Intakes Sorted by Site

fig4-1

While the intake records provide baseline information about the clients who enrolled in each of the programs, change in status can only be determined for those clients for whom we have both an intake record and a follow-up interview record. As indicated in Figure 4-2, there are 478 clients for whom no follow-up interview records exist; thus, we can only assess the change in status for 1,060 enrolled clients. Figure 4-2 also indicates the number of follow-up interview records available for each site. There is a noticeable difference in the percentage of enrolled clients lost to follow-up across locations: The highest rates of follow-up are for Fall River, where 41 of 286 clients (14 percent) were lost to follow-up, and among Union City youth, where 45 of 240 enrolled youth (or just under 19 percent) did not have follow-up interviews. Fall River clients received case management services, an intensive service delivery approach that generally resulted in a strong relationship between the staff member and the client. Few clients were lost under this delivery model. Union City youth, by contrast, were essentially a "captive audience" as they were required to participate in services to address substance use if they wanted to stay in school. Although quite a few youth did not have follow-up interviews, overall the retention rate in the program was very good.

FIGURE 4-2 Follow-Up Rates

fig4-2

Outcomes assessments for both Lorain and Union City adults are more challenging, with nearly 40 percent of enrolled clients in each site not participating in a follow-up interview. There are a couple of explanations for this. First, reportedly early on in each program, project staff enrolled clients into the program at the point at which they were screened and referred for services. Many of these individuals simply never followed through on the service referral and were lost to follow-up from the outset. Both programs subsequently altered their policies and formally enrolled clients only after they had actually participated in program services (Lorain discovered that many individuals showed up for one session and did not return, and thus began enrolling clients only after they showed up for a second visit). In addition, it was reported for Union City that the program requirements around completing follow-up interviews were unclear to staff early on, which resulted in a high rate of loss to follow-up. Once the issue was clarified, the follow-up rates for Union City adults improved significantly.

To assess the effectiveness of the services delivered under the CRRI grants, the SAIS dataset needs to contain an intake record (baseline), a follow-up interview (any change in baseline characteristics 6 months later), and a discharge record, which contains the information about the services each client received. As indicated in Figure 4-2, there are 813 clients for whom we have a complete set of records. In Fall River, the follow-up and discharge interviews appear to have occurred at the same 6-month point in time, so all 245 clients who have follow-up records also have discharge records. The other programs do not demonstrate this one-to-one correspondence. Among Union City adults, for example, 76 percent of clients with a follow-up interview also have a discharge record and nearly 82 percent of youth have all three records available for analysis. In Lorain, 53 percent of clients who have a follow-up interview also have a discharge record. There are several explanations for these disparities: First, and given the employment-related concerns of enrollees (or, for youth, their parents), it is quite possible that many individuals simply left the area in search of other work opportunities. Second, some of the individuals were discharged for failing to participate in their selected programs. Many of these clients might have been hard to find 6 months after their enrollment. Third, and for Lorain only, many clients who enrolled in the PRIDE program were eligible for more than one service. Westat was told that clients were not formally discharged from the program until they had availed themselves of all services for which they were eligible. Because there was no time limit for service receipt, clients who were still eligible for services simply were not discharged from the program. In addition, because PRIDE services were decentralized (i.e., the program was run by a board, not a service provider agency), staff reported that it was not always clear among providers which agency was responsible for completing the discharge interview. Thus Lorain's numbers likely under-represent the number of clients who benefited from PRIDE services; however, Westat is only able to conduct an assessment of their programs' effectiveness for those clients who had the full set of records.

View full report

Preview
Download

"CRRIfr.pdf" (pdf, 1.34Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®