HUD/HHS/VA Collaborative Initiative to Help End Chronic Homelessness
National Performance Outcomes Assessment

Is System Integration Associated with Client Outcomes?

June 15, 2007

By:
Alvin S. Mares, Ph.D., M.S.W. (1, 2, 3)
Greg A. Greenberg, Ph.D. (1, 2)
Robert A. Rosenheck, M.D. (1, 2, 4, 5)
  1. Northeast Program Evaluation Center, VAMC West Haven, CT
  2. Yale University Department of Psychiatry, New Haven, CT
  3. The Ohio State University College of Social Work
  4. VA New England Mental Illness Research, Education, and Clinical Center
  5. Yale University School of Epidemiology and Public Health, New Haven, CT

Address correspondence to

This report is available on the Internet at:
http://aspe.hhs.gov/hsp/homelessness/CICH07/integration/

Printer Friendly Version in PDF format (43 pages)

How to Obtain a Printed Copy

Contents

Acknowledgements

Executive Summary (in PDF format - 4 pages)

Sections:

References

Tables:

  1. Baseline characteristics of national evaluation subjects across sites
  2. Changes in CICH client service use and client outcomes
  3. Descriptive characteristics of network measures across sites
  4. Client characteristics bivariately associated with network measures
  5. Multivariate association of network measures and client measures


Acknowledgements

Peggy Halpern, Paul Dornan, Pete Dougherty, Anne Fletcher, Cynthia High, Mark Johnston, Robyn Raysor, and Gay Koerber participated in the Federal Funders CICH Evaluation Group representing HUD, DHHS, and VA provided essential oversight and review on earlier drafts of this paper.

We wish to acknowledge specifically the CICH evaluation site coordinators who coordinated data collection at their sites: Joyce Jones and Daniel White (Chattanooga), Eugene Herskovic (Chicago), JuanitaWilson (Columbus), Richard DiBlasio (Denver), Daniel Robbin and Elaine Stein (Ft. Lauderdale), John Nakashima (Los Angeles), Phyllis Larimore (Martinez), Julie Irwin (New York), Vincent Kane and Kimberly Lewis (Philadelphia), Lawrence Brennan (Portland), and Charlene Nason (San Francisco).

Brandi Williams coordinated data management at the VA Northeast Program

Evaluation Center (NEPEC) and Dennis Thompson provided computer programming support.


Introduction

Collaborative Initiative to Help End Chronic Homelessness (CICH)

In response to the goal of eliminating chronic homelessness, and the request that federal agencies increase their level of collaboration to accomplish this goal, the U.S. Departments of Housing and Urban Development (HUD), Health and Human Services (DHHS), and Veterans Affairs (VA) launched a new 3-year federal Initiative in October 2003 through the U.S. Interagency Council on Homelessness entitled The Collaborative Initiative to Help End Chronic Homelessness (CICH). Through this Initiative, persons experiencing chronic homelessness receive permanent supported housing funded by HUD, and supportive primary healthcare and mental health services provided by the Health Resources Services Administration (HRSA), the Substance Abuse and Mental Health Services Administrations (SAMHSA) of DHHS, and by the Veterans Health Administration (VHA) of VA. A chronically homelessness person was defined, in this initiative, as "an unaccompanied homeless individual with a disabling condition who has either been continuously homeless for 1 year or more or has had at least four episodes of homelessness in the past 3 years" (NOFA, 2003).

The key components of the CICH intervention involved: 1) providing comprehensive primary health, mental health, and substance abuse treatment services linked to housing; 2) creating additional permanent housing; 3) increasing the use of mainstream resources that pay for services and treatment for this population; 4) replicating service, treatment and housing models that have proven to be effective ( e.g. Stein & Test, 1980; Tsemberis & Eisenberg, 2000); and 5) supporting the development of infrastructures that sustain the housing, services and treatments and interorganizational partnerships beyond the designated CICH funding period.

The 11 communities funded by HUD, HHS and VA to implement CICH included Chattanooga, TN; Chicago, IL; Columbus, OH; Denver, CO; Ft. Lauderdale, FL; Los Angeles, CA; Martinez, CA; New York, NY; Philadelphia, PA; Portland, OR; and, San Francisco, CA.

Each of these communities (sites) developed a comprehensive plan to end or reduce the prevalence of chronic homelessness in their community through the development of sustainable, cost-effective partnerships among providers in the private and public sector. The specifics of these plans varied across communities, but each plan included strategies for providing permanent housing, linking comprehensive supports with housing, increasing the use of mainstream services; integrating systems and services, and, ensuring the sustainability of these efforts beyond the initial 3-year funding period.

National Performance Outcomes Assessment

The three federal agencies sponsoring the Initiative (HUD, DHHS & VA) enlisted the VA Northeast Program Evaluation Center (NEPEC) to conduct a national evaluation of CICH network collaboration and client outcomes to monitor the implementation and effectiveness of the $35-million Initiative by using a common evaluation methodology across all 11 sites.

Client outcomes evaluation

The goals of the client outcomes component of the national evaluation were to provide a site-by-site description of program implementation, as well as descriptive information on clients served; services received; longitudinal housing quality, stability, and satisfaction; and, client outcomes in health and functional domains. Outcome data from the evaluation were provided to the sites throughout the implementation of program to guide program development, and have been provided on a quarterly basis to the sites to allow ongoing monitoring of service delivery and outcomes. Monthly site-level statistics were provided to local CICH grant recipients on the implementation of evaluation procedures as well as to federal sponsors, beginning in May 2004, and updated longitudinal client outcome statistics were provided every 3 months beginning in January 2005. Preliminary outcomes data from the first 12-months of program operation were reported in early 2007 and showed improved client outcomes in housing and health status (Mares & Rosenheck, 2007).

Network collaboration evaluation

The network component of the national evaluation examined the extent to which local agencies collaborated in delivering CICH services along four dimensions: 1) implementing clinical best practices as recommended by SAMHSA; 2) coordinated service delivery and planning; 3) inter-organizational trust and respect; and, 4) exchanging resources. Annual assessments in each of these dimensions were made by telephone interviews administered by NEPEC staff to one key informant at each of the primary human service agencies delivering CICH services, or an average of 6.7 key informants/agencies per site. Three years of key informant interview data were collected around January of each year, beginning in 2004. Preliminary network outcome data from the first 3 years of network data collection were also reported in early 2007, and showed high levels of collaboration trust and increased use of evidence-based practices across the 11 sites over the period of program implementation (Greenberg & Rosenheck, 2007).

In this report, baseline network outcomes data were merged with 12-month client outcomes data to examine the association of inter-agency collaboration at the start of the program with client outcomes during the first year of program operation. The association between changes in inter-agency collaboration over time and longitudinal client outcomes will be examined after the collection of CICH client outcomes data is completed.

[ Go to Contents ]

Methods

Sample

CICH clients

During the first two years of program operation (i.e., through April 2006), 1,430 homeless people were formally screened for enrollment in CICH. Of these, 1,242 (87%) were identified as having been enrolled into the CICH clinical program. All of those enrolled into CICH were invited to participate in the national evaluation. Among the 1,242 enrolled into the program nationally, 734 (59%) gave written informed consent to participate in the national evaluation. All those who provided written informed consent participated in data collection for the national evaluation. Participation in the national evaluation was completely voluntary, and did not influence receipt of housing or services provided through the Initiative. Informed consent procedures were approved by Institutional Review Boards (IRBs) at the VA Connecticut Healthcare System (where the Northeast Program Evaluation Center is located) and Yale University (at which the investigators have faculty appointments), and at each of the 11 local VA medical centers through which local evaluation data were collected by VA research staff and affiliated academic institutions where appropriate.

Key informants

Soon after the three federal agencies sponsoring CICH announced the selected sites , in fall 2003, "lead agency" staff responsible for administering CICH funds and "partner agency" staff providing services to CICH clients at each of the 11 CICH sites were asked to identify up to ten "core" human service agencies that made up their local network of CICH service providers. Included in each network were the lead agency which was primarily responsible for implementing the program and coordinating the actions of partnering agencies, and those partnering agencies primarily responsible for providing housing assistance, mental health care, substance abuse services, primary health care, and Veterans Health Administration services to CICH clients. Key informants were those identified by program leaders as those most knowledgeable about the activities of each network agency at each site.

In some sites one agency provided more than one key service (e.g., the mental health agency also provided substance abuse services); thus, only one survey was conducted for that agency. At other sites, in contrast, more than one agency provided a particular service and key informants at both agencies were interviewed. Thus, the number of agencies at each site ranged from five to nine. Furthermore, at some agencies more than one key informant was identified and interviews were jointly held.

Data collection

Client data

Client data were collected by full-time VA employees serving as "evaluation assistants," one at each site. These evaluation assistants were responsible for recruiting participants, collecting screening/intake forms completed by case managers, and administering baseline assessment interviews at entry into the formal program evaluation, along with quarterly follow-up assessment interviews for up to 3 years. Follow-up interviews were administered regardless of clients' housing or treatment status i.e., evaluation assistants continued the administration of follow-up interviews to clients who remained engaged in treatment, as well as those who discontinued participation in either housing or case management services throughout the Initiative.

Data collection began at the start of program initiation at each site, from March - August 2004, and is intended to continue through September 2007. The data presented in this report were collected from the start of the project through mid-May, 2006, and represent 97% of all anticipated 3-month data, 93% of 6-month data, 88% of 9-month data, and 81% of all 12-month data to be collected. Thus we have limited 3-year data, but have nearly complete 1-year outcome data that have been collected and are available for presentation in this preliminary report.

Network data

After the initial "network definition" survey was completed at each of the 11 sites and the network sampling frame consisting of core agencies and key informants was defined, a second, more extensive "network participation survey" was then administered in three waves, between November 2003 to March 2006, to the key informants identified at each of the participating agencies. The first survey wave of data collection occurred before CICH began, between November 2003 and March 2004. The second and third waves of network data collection occurred at the end of the first and second years of program operation (i.e. from November 2004 to February 2005, and from January to March 2006).

Among these three annual network surveys, one was classified as the "current" survey for each client based upon the date that he/she was enrolled into the national evaluation. The specific method used for doing so is described in the "Methods — Merging of Client and Network Data" section of this report.

Network participation surveys were sent to all key informants prior to the interviews, which were conducted over the telephone by national evaluation staff at the Northeast Program Evaluation Center.

Measures

A brief summary of client and network measures are provided below. More detailed descriptions of these measures have been reported elsewhere (Mares & Rosenheck, 2007; Greenberg & Rosenheck, 2007).

Client measures

Individual characteristics

CICH intake and outreach staff completed a structured form on each person screened, which documented: 1) basic socio-demographic information (i.e., age, gender, and race/ethnicity); 2) eligibility characteristics (i.e., single individual vs. parent, and type of chronic homelessness experience — a current episode of homelessness lasting a year or longer vs. 4 or more episodes of homelessness during the past 3 years); 3) each of three disabling condition(s) (i.e., mental health, substance abuse or medical) identified at screening and during the baseline assessment interview; 4) outreach location and the agency initiating outreach; 5) outreach clinician observations of clinical problems; 6) response to early interactions with intake/outreach staff (i.e. interest in participating in the program); and, 7) date enrolled into the program (if applicable). Measures of time (days) from screening to enrollment (among those enrolled into the program), and time from enrollment to the baseline assessment interview (among CICH clients) were also documented.

Supplemental socio-demographic and clinical information (e.g., veteran status, disabling condition(s)) were collected during the baseline assessment interview, along with service use and client outcome data.

Service Use
Primary health care services

First, CICH clients were asked whether they had a "usual health care provider", and whether they had health insurance?

Then the number of routine, preventive healthcare procedures received during the past year was assessed from a list of 12 gender-neutral procedures (e.g., measurement of height, weight, blood pressure, cholesterol), plus either 4 male-specific procedures (e.g., prostate exam, PSA testing) or 2 female-specific procedures (i.e., PAP testing and breast exam) (Heslin, Andersen & Gelberg, 2003).

Clients reporting unhealthy behaviors were also asked the number of such behaviors that they had discussed with a healthcare professional during the previous year. These behaviors included drinking alcohol among drinkers, smoking among smokers, and diet/nutrition among those who were obese at baseline.

The total number of outpatient medical visits made during the past 3 months was also included as a primary health treatment measure.

Finally, the trust in physician scale (Anderson & Dedrick, 1990), an 11-item measure, was used to assess the level of trust felt by a patient with his/her primary doctor/physical healthcare provider.

Mental health services and substance abuse treatment

Clients were further asked whether they could identify a primary mental health or substance abuse treatment provider, as well as their total number of outpatient mental health visits and the total number of outpatient substance abuse treatment visits during the previous 3 months. A fourth measure addressed participation in self-help groups (i.e., Alcoholics Anonymous or Narcotics Anonymous)

A 7-item therapeutic alliance scale was used to measure the strength of the therapeutic relationship experienced by CICH clients with their primary mental health or substance abuse provider (Neale & Rosenheck, 1995).

Finally, clients' experience of personal choice in selecting mental health or substance abuse services was measured using a 5-item "consumer choice" scale (Monahan et. al., 2005).

Case management

Clients were also asked whether they could identify a primary case manager, and whether they were visited by a case manager in a community setting (i.e., either at home or at some other place in the community other than a service agency or healthcare facility setting) during the previous 3 months.

Clients were further asked whether they had a money manager ("a person or organization which helps you manage your money") and whether they had had any contact with their landlord, either in-person or by telephone, during the past 3 months? Money management has been identified as an important ingredient in the approach to helping homeless people developed by Tsemberis and colleagues (2000) - and landlord-tenant relationships have been found to be associated with housing outcomes among persons with mental health and/or substance abuse problems (Kloos et. al., 2002).

Services integration

Both objective and subjective measures were used to evaluate the integration and coordination of diverse CICH services. Services integration was defined as the extent to which the key components characterizing the CICH intervention (listed below) were provided to clients. An objective measure of overall services integration was based on calculation of the proportion of total component services received by clients using a series of six dichotomous service component measures including: 1) independent housing 2) case management, 3) general medical care, 4) substance abuse treatment, 5) mental health services, and 6) VA services. A higher score on this overall measure represented a more fully integrated service delivery.

A second objective measure of overall service delivery represented the total number of outpatient health visits of all kinds — including medical, mental health, and substance abuse treatment visits — received during the previous 3 months. Clients were further asked to estimate the total number of different individual service providers assisting them during the past 3 months.

To supplement these objective measures, a 5-item subjective scale was developed to measure the extent to which the delivery of these services was perceived to be well coordinated or fragmented. Higher scores on this measure reflected a higher degree of coordination of services and less fragmentation.

Client outcomes
Housing

Clients were asked at each interview the number of days during the past 3 months that they were housed in each of nine settings, as well as where they were residing at the time of each assessment. The number of days "housed" was defined as living in their own place, someone else's place, or in an SRO hotel or boarding home. SRO hotels were considered residences because some sites used such housing as the primary housing resource for CICH clients at some sites (i.e., Los Angeles, San Francisco). Nights spent in shelters, outdoors, in vehicles, or in abandoned buildings were classified as representing "homeless" housing status.

Clients who were living in their own places were also asked to report their level of satisfaction with their housing using a 20-item housing satisfaction scale developed by Tsemberis and colleagues (2003), as part of the SAMHSA Supported Housing Initiative (CMHS, 2001).

Community adjustment

To evaluate how well CICH clients were integrated into and engaged in community life, they were asked whether they had participated in each of 16 common activities (e.g., visiting with others, going to a grocery store, reading a newspaper) during the previous 2 weeks (Katz, 1963). Responses to these items were then summed to create a "community involvement" scale measure, where a higher score represented greater participation in community activities.

Social support networks were assessed by questions asking the number of types of persons who would be available to help them about three different types of assistance: a short-term loan of $100, a ride to an appointment, or someone to talk with if they felt suicidal (Vaux et. al., 1987).

Additional single-item measures addressed a) whether clients knew any of their neighbors well, b) the number of days spent in jail during the past 3 months, and c) satisfaction with life overall (subjective quality of life), scored on a 7-point terrible=1 to delighted=7 scale, with a higher score indicating greater satisfaction with life. (Lehman, 1988).

Income

CICH client income was expected to rise as a result of participating in the program, both through increased access to public support benefit payments and through employment. Clients were asked whether they had received any of several types of public support income during the past month, and if so, the amount of such income. Information on days of employment and employment income were also obtained, along with informal types of income. Responses to these items were summed to create a measure of total income.

Mental health and physical health status

The Medical Outcomes Study Short Form (SF)-12 mental health subscale (Ware et. al., 1998), three subscales from the Brief Symptom Inventory (BSI) (Derogatis & Spencer, 1982), and an observed psychotic behavior rating scale (Dohrenwend, 1982) were used to evaluate mental health status.

Substance abuse

Items from the Addiction Severity Index (ASI) documented the number of days each client drank to intoxication and whether they had used any illicit drugs during the previous month. Alcohol and drug sub-scales (McLellan et. al., 1980) measured alcohol and drug use problems. A higher score on an these ASI sub-scales reflect a greater, more serious substance use problem.

Service costs

Service costs were estimated for four aggregated types of care: medical/dental treatment, mental health services, substance abuse services, and the total for all three types of services. Costs were estimated on the basis of average unit cost data compiled for a recent NIMH funded cost effectiveness study of treatment of schizophrenia (Rosenheck et. al., 2006).

Total health costs were also sub-grouped across types of service into inpatient and outpatient costs.

Network measures

Best practices

An evidence-based practices scale was used to measure the degree to which each of the 11 CICH networks was viewed by its core agencies as having implemented evidence-based practices, which included 18 practices identified in 2003 by the Substance Abuse and Mental Health Services Administration (SAMHSA) (SAMHSA, 2003). A higher score on this scale represented a greater use of evidence-based practices.

Services planning

A services planning scale was developed to represent inter-agency joint planning and coordination, which included cooperation in serving clients, goal congruence, client referral, cooperative planning, co-location of staff and services, information sharing and communication.

Trust and respect

A third scale assessed the degree to which trust and respect existed between agency dyads. A higher score on this scale represented a higher level of trust and respect.

Exchange of resources

An exchange of resources scale was constructed to measure whether an active fiscal relationship existed between each pair of agencies at each site. For every pair of agencies this scale indicated whether there existed any transfer of resources.

Merging of Client and Network Data

Individuals were recruited into CICH, and subsequently into the evaluation, on a rolling-basis throughout the first two years of program operation. Baseline assessment interviews were administered as soon as possible after clients entered the program and provided written informed consent to participate in the evaluation. Follow-up assessments were then administered as close to 90, 180, 270, and 360 days after the date of the baseline assessment for 3-month, 6-month, 9-month, and 12-month follow-up's, respectively. Thus, the client outcome data reported here were collected on a rolling-basis throughout the first three years of program operation (i.e., recruitment throughout years 1 and 2 of the program, followed by 4 quarterly follow-up assessments).

Network data, by contrast, were collected annually for all sites around January of each year, beginning in January 2004, approximately 3 months after the sites were selected and 3 months before the programs became operational (i.e., during the 6-month post-selection implementation planning phase of the program).

Client outcomes data were merged with network data to most accurately reflect the status of network collaboration during their first year in treatment; namely, clients whose baseline assessment occurred before 7/1/04 (N=279, 40%), between 7/1/04 and 7/1/05 (N=393, 56%), and after 7/1/05 (N=28, 4%) were merged with corresponding year 1, year 2, and year 3 network data, respectively. These dates represent the approximate mid-points between network interviews.

This method was used for selecting a single "current" network interview from among three potential network interviews to minimize the potentially confounding factor of changes in network operation over time. Previous analyses of network data mentioned earlier showed significant changes among network measures over the three waves of network data collection (Greenberg & Rosenheck, 2007). Given a) the relationship between "current" network characteristics and first year client outcomes examined in this report, b) the early administration of the first network interview prior to the start of clinical program operation, and c) that clients were enrolled into the clinical program (and thus into the national evaluation) staggered over a two-year period of time, the decision was made to use the network interview that most reflected the status of the service network at the time that each client was enrolled into the national evaluation. Network characteristics at the time of client entry into the program were judged to be most pertinent to that consumer's overall experience of the service system at their site.

Statistical Analysis

Bivariate correlations between 142 socio-demographic and health status characteristics of clients at baseline and the 4 network measures were used to identify potentially confounding client characteristics to be included as covariates in subsequent multivariate analyses (Table 4).

Next, four sets of repeated measures mixed regression models were used to examine the association of each network measure with 16 client service use measures and 18 client outcome measures during their first year in CICH (Table 5). Prior multivariate analyses showed significant changes over time among these 34 client service use and outcome measures during the first year of program participation (Mares & Rosenheck, 2007; Table 2).

The main effects of time and status on each of the four network measure (independent variables) were examined, covarying for the baseline value of each client service use or client outcome measure (dependent variable) and additional client baseline characteristics bivariately associated with each network measure.

Thus, a total of 136 mixed regression models were used evaluating the relationship between 4 network independent variables and each of 34 client dependent variables (4*34= 136)(Table 5). Statistical significance for network measures are presented in this report, along with coefficients for those network measures found to be significantly associated with client service use and outcomes measures at the p<.05 level of significance.

Significant results at the Bonferroni-adjusted p<.001 were highlighted in bold type. The Bonferroni correction corrects for multiple-comparisons when several statistical tests are being performed. The smaller p<.001 level of significance was calculated by dividing the conventional p<.05 level of significance by 34, the number of dependent variables tested (i.e., .05/34=.001).

[ Go to Contents ]

Results

CICH Client Characteristics

At the time of program entry, CICH clients had been homeless an average of 8 years in their lifetimes; 72% had substance abuse problems; 76% had mental health problems, and 66% reported medical problems (Table 1). Other client characteristics are presented in Table 1. All but ten clients (98.6%) reported having at least one of the four qualifying disabilities for the project (i.e., substance abuse, mental health, or medical problem, or a developmental disability).

Changes in Service Use and Outcomes

Of the six core services targeted for CICH clients, the proportion of these services received by individual clients, including both housing and healthcare services, increased from an average of 64% at baseline to 78-81% during the following 12 months (Table 2).

The average number of days housed in the previous 90 days increased dramatically from 18 at baseline, to 68 at the 3-month follow-up, and rose steadily thereafter to 83 at the 12 month follow-up (Table 2). Mean monthly public assistance income increased steadily from $316 at baseline to $478 one year later, a 50% increase. Significant improvements of modest magnitude were also observed in overall quality of life, mental health functioning, and reduced psychological distress. Alcohol and drug problems remained largely unchanged over time. Total quarterly health costs declined by 50%, from $6,832 at baseline to $3,376 at 12 months. A 54% decrease in mean inpatient costs ($5,776 to $2,677) accounted for nearly 90% of the overall decrease in quarterly health care costs during clients' first year in the program (Table 2).

CICH Network Characteristics

Descriptive statistics on the four network measures attributed to CICH clients are presented in Table 3. Means and ranges for each measure are provided for each site and for the total CICH sample.

Client Characteristics Bivariately Associated with Network Characteristics

The number of baseline client characteristics bivariately associated with network measures included 23 for the use of evidence-based practices network measure, 39 for the services co-planning and integration, 41 for the exchange of resources, and 43 for the trust measure (Table 4). These significant characteristics were included as covariates in the multivariate findings summarized below.

Multivariate Association of Network Characteristics and Changes in Client Outcomes

Service use

After adjusting for baseline covariates, and the baseline value for each client service use measure, significant associations at p<.05 were found in 10 of 64 network-client service use relationships examined. The network measure associated with the largest number of measures of client service use was the implementation of best practices. While previous analyses of CICH service use data showed increasing use of preventive primary health care services, discussion of healthcare behaviors with one's primary care provider, and receipt of case management visits in the community (Table 2), use of all three of these types of services were found, paradoxically, to be lower among clients served by networks that reported implementing more evidence-based practices (Table 5, Best Practices data column). Clients served by networks implementing best practices to a greater degree were more likely to receive money management services, but to report being treated by fewer different service providers. Networks more involved in co-planning and integration of services showed greater increases in the proportion of CICH clients who reported having a usual provider for both physical health and mental health problems over time (1.5, p<.05; and, 2.5, p<.01, respectively) (Table 5, Services Co-Planning data column).

Clients served by networks which exchanged resources to a greater extent were also more likely to have a primary treater for mental health over time (0.59, p<.05), but were less likely to receive case management visits in the community over time (-0.60, p<.05) (Table 5, Exchange Resources data column).

Finally, clients served by networks with higher levels of interagency trust were more likely to show increasing proportions of clients with health insurance over time clients (-0.31 uninsured, p<.05)

Client Outcomes

Significant associations at p<.05 were found on only 2 of the 72 network-client outcome relationships, and among only 1 of the 4 network measures. Clients served by networks implementing more evidence-based practices expressed more trust in their primary healthcare provider (0.24, p<.05), and had lower total treatment costs (-2260, p<.05), than clients treated in networks reporting lower levels of implementation of best practices (Table 5, Best Practices data column).

At the Bonferroni-adjusted p<.001 level of significance the only significant relationship between network measures and either client service use or outcomes was the positive relationship between increased implementation of best practices and increased likelihood of receiving money management services (Table 5).

[ Go to Contents ]

Discussion

No significant relationship was found between service network characteristics and changes in client outcomes, except for an increase in satisfaction with the primary healthcare provider and a decrease in total service costs among clients living in networks utilizing more evidence-based practices.

There were also few significant associations found between network characteristics and changes in clients service use, and where significant associations were found, they were often opposite to the expected direction. Most notably, clients served by networks using more evidence-based practices received fewer preventive healthcare procedures, fewer case management visits in the community, and they were cared for by a fewer number of service providers than clients served by other networks.

The only Bonferroni-adjusted finding of significance, adjusting for multiple comparisons, was that clients served by best practices networks were more likely to receive money management services. Thus, the primary conclusion of this report is that there were few significant relationships between network characteristics and either client service use or client outcomes.

One possible explanation for the overall lack of significant finding is that the level of inter-agency collaboration at the start of the project was too high to differentially impact changes in clients' use of services and outcomes over time. Mean network measure scores were relatively high on three of the four measures examined — i.e., mean scores ranged from 1.8 to 2.7 (out of a possible score of 3.0) (Table 3).

Another possible explanation is that there was a lack of variability across sites on either network measures and/or client measures. If either type of measure was constant across sites, then no significant relationships between the two would be found. This was not, however, the case since previous client-level analyses found significant statistical differences across sites in outcomes on all but one client measure (Mares & Rosenheck, 2007). Statistically significant differences were also found across sites on all network measures (Table 3, last row). The variability among sites ranged from a low of 14% (2.45-2.80) for trust and 26% (2.08-2.63) for implementation of best practices to 59% (1.31-2.09) for services planning and coordination and 1,475% (0.04-0.63) for the exchange of resources. This degree of variability among measures of the trust and best practices may have been insufficient to result in programmatically distinguishable differences across sites at the client level.

A fourth explanation is that network collaboration is not actually associated with client outcomes  — at least not initial levels of network collaboration at the start of the CICH program. System characteristics may be too remote from client experience to affects individual service use or outcomes.

Subsequent analyses may yet find a significant association between network collaboration and longer-term client service use and outcomes (i.e., with 3-year client outcomes data that are still being collected at the present time) It is also possible that changes in network measures over the full 3-year program period will be found to be significantly associated with longitudinal client service use and outcomes measures.

The overall lack of significant findings is not altogether surprising, however, given that two previous evaluations of system integration initiatives found no substantial improvement in treatment outcomes at the individual client level with greater levels of integration. In 1986 the Program on Chronic Mental Illness supported by Robert Wood Johnson and HUD to evaluate whether more highly integrated systems of care were more effective in addressing the needs of persons with serious mental illness, found that integration efforts were associated with increased inter-agency collaboration and increased continuity of care, but were not associated with improved client outcomes, such as symptoms, social relationships, and quality of life (Morrissey et. al., 1994; Lehman et. al., 1994).

Similar results were found in a study by the Center for Mental Health Services in the Access to Community Care and Effective Services and Supports (ACCESS) program. Implemented in 1993, this 5-year demonstration program examined the impact of efforts to enhance system integration on outcomes of homeless persons with serious mental illness (Cocozza et. al., 2000; Rosenheck et. al., 2002), and included a matched sample of comparison sites that did not implement integration strategies. While demonstration sites were found to have more integrated service systems (Morrissey et. al., 2002), clients at these sites showed no greater improvement in clinical health status, or 12-month housing outcomes than at control sites (Rosenheck et al., 2002), although housing outcomes were more favorable among clients served in more integrated service systems, regardless of intervention status (Rosenheck et. al., 1998).

Limitations

The major limitation of this preliminary report is the self-report nature of both client and network-level data. While substantial efforts were made to develop accurate survey instruments and standardized administration of both client and key informant interviews, validation of these self-report data were not possible as part of this project.

Conclusion

These preliminary findings suggest that collaboration among CICH network agencies at the start of the program was not significantly associated with either client service use nor client outcomes during clients' first year of entering the program.

Questions regarding the association of changes in network collaboration and client service use and outcomes over the entire 3-year program follow-up period will be addressed in a subsequent report (around January 2008) after CICH client data collection is completed in fall 2007.

[ Go to Contents ]

References

Anderson, L.A. & Dedrick, R.F. (1990). Development of the Trust in Physician Scale: A measure to assess the interpersonal trust in patient-physician relationships. Psychological Reports, 67:1091-1100.

Center for Mental Health Services. (2001). The CMHS housing initiative. Rockville, MD: Substance Abuse and Mental Health Services Administration.

Cocozza, J.J., Steadman, H.J., Dennis, D.L., Blasinsky, M., Randolph, F.L., Johnsen, M. & Goldman, H. (2000). Successful systems integration strategies: The ACCESS program for persons who are homeless and mentally ill. Administration and Policy in Mental Health 27(6):395-407.

Derogatis, L.R. & Spencer, N. (1982). The brief symptom index; Administration, scoring and procedure manual, Baltimore MD, Johns Hopkins.

Dohrenwend, B. (1982). Psychiatric Epidemiology Research Interview (PERI). New York: Columbia University Social Psychiatry Unit.

Greenberg, G.A. & Rosenheck, R.A. (2007). An evaluation of an initiative to improve coordination and service delivery of homeless services networks. West Haven, Connecticut: VA Northeast Program Evaluation Center. February 13, 2007.

Heslin, K.C., Andersen, R.M., & Gelberg, L. (2003). Case management and access to services for homeless women. Journal of Health Care for the Poor and Underserved, 14(1):34-51.

Katz, A.H. (1963). Social adaptation in chronic illness: A study of hemophilia. American Journal of Public Health and the Nation's Health, 53(10):1666.

Kloos, B., Zimmerman, S.O., Scrimenti, K., & Crusto, C. (2002). Landlords as partners for promoting success in supported housing: "It takes more than a lease and a key." Psychiatric Rehabilitation Journal, 25(3):235-244.

Lehman A. (1988). A quality of life interview for the chronically mentally ill. Evaluation and Program Planning, 11:51-62.

Lehman, A.F., Postrado, L.T., Roth, D., McNary, S.W. & Goldman, H.H. (1994). Continuity of care and client outcomes in the Robert Wood Johnson Foundation Program on Chronic Mental Illness. Milbank Quarterly 72(1):105-122.

Mares, A.S. & Rosenheck, R.A. (2007). HUD/HHS/VA Collaborative Initiative to Help End Chronic Homelessness national performance outcomes assessment: Preliminary client outcomes report. West Haven, Connecticut: VA Northeast Program Evaluation Center. February 23, 2007.

McLellan, A., Luborsky, L., Woody, G, et al. (1980). An improved diagnostic evaluation instrument for substance abuse patients: the Addiction Severity Index. Journal of Nervous and Mental Disease, 168:26-33.

Monahan, J., Redlich, A.D., Swanson, J.R., Clark, P., Appelbaum, P.S., Petrila, J., Steadman, H.J., Swartz, M., Angell, B., & McNiel, D.E. (2005). Use of leverage to improve adherence to psychiatric treatment in the community. Psychiatric Services, 56(1):37-44.

Morrissey, J.P, Calloway, M. et. al. (1994). Local mental health authorities and service system change: Evidence from the Robert Wood Johnson Foundation Program on Chronic Mental Illness. Milbank Quarterly 72(1).

Neale, M.S. & Rosenheck, R.A. (1995). Therapeutic alliance and outcome in a VA intensive case management program. Psychiatric Services, 46(7):719-723.

NOFA. (2003). Notice of funding availability for the Collaborative Initiative to Help End Chronic Homelessness. Federal Register, Vol. 68, No. 17, January 27, 2003.

Rosenheck RA, Leslie D, Sindelar J, Miller EA, Lin H, Stroup S, McEvoy J, Davis S, Keefe RSE, Swartz M, Perkins D, Hsiao J, Lieberman JA (2006). Cost-Effectiveness of Second Generation Antipsychotics and Perphenazine in a Randomized Trial of Treatment for Chronic Schizophrenia. American Journal of Psychiatry; 163(12):2080-2089.

Rosenheck, R.A., Morrissey, J., Lam, J., Calloway, M., Johnsen, M. Goldman, H.H., Calsyn, R., Teague, G., Randolph, F., Blasinsky, M. & Fontana, A. (1998). Service system integration: Access to services and housing outcomes in a program for homeless persons with severe mental illness. American Journal of Public Health 88(11):1610-1615.

Rosenheck, R.A., Morrissey, J., Lam, J., Calloway, M., Stolar, M., Johnsen, M., Randolph, F., Blasinsky, M. & Goldman, H.H. (2001). Service delivery and community: Social capital, service systems integration, and outcomes among homeless persons with severe mental illness. Health Services Research 36(4):691-710.

Rosenheck RA, Lam J, Morrissey JP, Calloway M, Marilyn Stolar, Randolph F, Blasinsky M, Johnsen, M, Steadman H, Cocozza J, Dennis DD, Goldman HH. Do Efforts to Improve Service Systems Integration Enhance Outcomes for Homeless Persons with Serious Mental Illness? Evidence from the ACCESS Program. Psychiatric Services. 2002:53(8):958-966.

SAMHSA. (2003). Blueprint for change: Ending chronic homelessness for persons with serious mental illnesses and co-occurring substance use disorders. DHHS Pub. No. SMA-04-3870, Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration.

Stein, L.I. & Test, M.A. (1980). Alternative to mental hospital treatment: I. Conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry, 37(4):392-397.

Tsemberis, S. & Eisenberg, R.F. (2000). Pathways to housing: supported housing for street-dwelling homeless individuals with psychiatric disabilities. Psychiatric Services, 51(4):487-493.

Tsemberis, S., Rogers, E.S., Rodis, E., Dushuttle, P., & Skryha, V. (2003). Housing satisfaction for persons with psychiatric disabilities. Journal of Community Psychology, 31(6):581-590.

Vaux, A. & Athanassopulou, M. (1987). Social support appraisals and network resources. Journal of Community Psychology, 15:537-556.

Ware, J.E., Kosinski, M., & Keller, S.E.. (1998). How to score the SF-12 physical and mental health summary scales, 3rd edition. Lincoln, Rhode Island: Quality Metric Inc.

[ Go to Contents ]


How to Obtain a Printed Copy

To obtain a printed copy of this report, send the title and your mailing information to:

Human Services Policy, Room 404E
Assistant Secretary for Planning and Evaluation
U.S. Department of Health and Human Services
200 Independence Ave, SW
Washington, DC 20201

Fax:  (202) 690-6562

You may just print the PDF version (43 pages)


Address correspondence to:

Alvin S. Mares, Assistant Professor
College of Social Work, The Ohio State University
325Q Stillman Hall
1947 College Rd.
Columbus, Ohio 43210
Phone: (614) 292-0425 / Fax: (614) 292-6940 / E-mail: mares.2@osu.edu


Where to?

Top of Page | Contents

Home Pages:
Evaluation of the CICH: Project Page
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)

Last updated:  03/10/08