No significant relationship was found between service network characteristics and changes in client outcomes, except for an increase in satisfaction with the primary healthcare provider and a decrease in total service costs among clients living in networks utilizing more evidence-based practices.
There were also few significant associations found between network characteristics and changes in clients service use, and where significant associations were found, they were often opposite to the expected direction. Most notably, clients served by networks using more evidence-based practices received fewer preventive healthcare procedures, fewer case management visits in the community, and they were cared for by a fewer number of service providers than clients served by other networks.
The only Bonferroni-adjusted finding of significance, adjusting for multiple comparisons, was that clients served by best practices networks were more likely to receive money management services. Thus, the primary conclusion of this report is that there were few significant relationships between network characteristics and either client service use or client outcomes.
One possible explanation for the overall lack of significant finding is that the level of inter-agency collaboration at the start of the project was too high to differentially impact changes in clients' use of services and outcomes over time. Mean network measure scores were relatively high on three of the four measures examined i.e., mean scores ranged from 1.8 to 2.7 (out of a possible score of 3.0) (Table 3).
Another possible explanation is that there was a lack of variability across sites on either network measures and/or client measures. If either type of measure was constant across sites, then no significant relationships between the two would be found. This was not, however, the case since previous client-level analyses found significant statistical differences across sites in outcomes on all but one client measure (Mares & Rosenheck, 2007). Statistically significant differences were also found across sites on all network measures (Table 3, last row). The variability among sites ranged from a low of 14% (2.45-2.80) for trust and 26% (2.08-2.63) for implementation of best practices to 59% (1.31-2.09) for services planning and coordination and 1,475% (0.04-0.63) for the exchange of resources. This degree of variability among measures of the trust and best practices may have been insufficient to result in programmatically distinguishable differences across sites at the client level.
A fourth explanation is that network collaboration is not actually associated with client outcomes at least not initial levels of network collaboration at the start of the CICH program. System characteristics may be too remote from client experience to affects individual service use or outcomes.
Subsequent analyses may yet find a significant association between network collaboration and longer-term client service use and outcomes (i.e., with 3-year client outcomes data that are still being collected at the present time) It is also possible that changes in network measures over the full 3-year program period will be found to be significantly associated with longitudinal client service use and outcomes measures.
The overall lack of significant findings is not altogether surprising, however, given that two previous evaluations of system integration initiatives found no substantial improvement in treatment outcomes at the individual client level with greater levels of integration. In 1986 the Program on Chronic Mental Illness supported by Robert Wood Johnson and HUD to evaluate whether more highly integrated systems of care were more effective in addressing the needs of persons with serious mental illness, found that integration efforts were associated with increased inter-agency collaboration and increased continuity of care, but were not associated with improved client outcomes, such as symptoms, social relationships, and quality of life (Morrissey et. al., 1994; Lehman et. al., 1994).
Similar results were found in a study by the Center for Mental Health Services in the Access to Community Care and Effective Services and Supports (ACCESS) program. Implemented in 1993, this 5-year demonstration program examined the impact of efforts to enhance system integration on outcomes of homeless persons with serious mental illness (Cocozza et. al., 2000; Rosenheck et. al., 2002), and included a matched sample of comparison sites that did not implement integration strategies. While demonstration sites were found to have more integrated service systems (Morrissey et. al., 2002), clients at these sites showed no greater improvement in clinical health status, or 12-month housing outcomes than at control sites (Rosenheck et al., 2002), although housing outcomes were more favorable among clients served in more integrated service systems, regardless of intervention status (Rosenheck et. al., 1998).