HUD/HHS/VA Collaborative Initiative to Help End Chronic Homelessness: An Evaluation of an Initiative to Improve Coordination and Service Delivery of Homeless Services Networks

By:
Greg A Greenberg, PhD
Northeast Program Evaluation Center, VAMC West Haven, CT,
Yale University Department of Psychiatry, New Haven, CT
and
Robert A Rosenheck, MD
Northeast Program Evaluation Center, VAMC West Haven, CT,
VA New England Mental Illness Research, Education, and Clinical Center;
Yale University Department of Psychiatry and School of Epidemiology and Public Health, New Haven, CT

 

Acknowledgements

Peggy Halpern, Paul Dornan, Pete Dougherty, Anne Fletcher, Cynthia High, Mark Johnston, Robyn Raysor, and Gay Koerber participated in the Federal Funders CICH Evaluation Group representing HUD, DHHS, and VA provided essential oversight and review on earlier drafts of this paper.

We wish to acknowledge specifically the CICH evaluation site coordinators who coordinated data collection at their sites: Joyce Jones and Daniel White (Chattanooga), Eugene Herskovic (Chicago), JuanitaWilson (Columbus), Richard DiBlasio (Denver), Daniel Robbin and Elaine Stein (Ft. Lauderdale), John Nakashima (Los Angeles), Phyllis Larimore (Martinez), Julie Irwin (New York), Vincent Kane and Kimberly Lewis (Philadelphia), Lawrence Brennan (Portland), and Charlene Nason (San Francisco).

Brandi Williams coordinated data management at the VA Northeast Program

Evaluation Center (NEPEC) and Dennis Thompson provided computer programming support.

Address Correspondence to:

Greg Greenberg
Northeast Program Evaluation Center
950 Campbell Ave
West Haven CT 06516
Tel.: (203) 937-3850
Fax: (2 03) 937-3433
E-Mail: Greenberg@biomed.med.yale.edu

Introduction

Enhancing service system integration has long been an objective of policymakers, payers, and human service providers. As far back as the Model Cities Program of the 1960s processes that facilitate system integration have been thought to improve the capacity of systems to address the needs of individuals with multiple problems and have been widely believed to increase the accessibility, continuity and coordination of care (Morrissey et al. 1994; Provan & Milward 1995; Randolph et al 2002; Cocozza et al. 2000; Foster-Fishman et al. 2001; Provan & Milward 1995). Emphasis on implementation of evidence-based practices has been a growing and parallel trend in health care (Guyatt et al. 2004) and has been rigorously promoted in several recent policy summaries of literature on the treatment of mental illness (Drake et al. 2001; Torrey et al. 2001; DHHS 1999; New Freedom Commission on Mental Health 2003; SAMHSA 2003). It has been suggested that more integrated systems may also allow for more rapid dissemination of evidence-based practices, with greater model fidelity⁽²⁾ (McKinney et al 1993; Goldman et al. 2001).

There is broad consensus that increasing the level of service system integration should lead to improved client outcomes. System integration can be defined broadly as the provision of services with high levels of coordination, communication, trust, and respect among service agencies so that they are better able to work together to achieve common objectives. While there is some supporting evidence from cross sectional data (Provan & Milward 1995; Rosenheck et al. 1998; Rosenheck, Morrisey, Lam, et al. 2001), two large prospective evaluations of system integration initiatives found that even when the levels of integration were increased there was no substantial improvement in treatment outcomes at the individual client level. One explanation for this finding may be that improved system integration may be necessary but not sufficient to improve client outcomes and that concomitant implementation of evidence-based clinical practices may also be necessary (Isett and Morrissey 2006; Goldman et al. 2001; Lehman et al. 1994).

In 1986, the Program on Chronic Mental Illness launched by the Robert Wood Johnson Foundation and the U.S. Department of Housing and Urban Development sought to evaluate whether more highly integrated systems of care were more effective in addressing needs of persons with severe mental illness (SMI). Results of this nine-site evaluation found that integration efforts were associated with measurable improvements in inter-agency collaboration and increased continuity of care (in part as a result of greater availability of case management services) but were not associated with improved client outcomes, such as symptoms, social relationships, and quality of life (Morrissey et al 1994; Lehman et al. 1994).

Similar results were reported from the 18-site evaluation of the Center for Mental Health Service's Access to Community Care and Effective Services and Supports (ACCESS) program. This five-year demonstration program, implemented in 1993, evaluated the impact of efforts to enhance system integration on outcomes of homeless persons with serious mental illness (Cocozza et al. 2000; Rosenheck et al. 2001) and represented a major improvement over the evaluation design of the Program on Chronic Mental Illness by including a matched sample of comparison sites that did not implement integration strategies. While intervention sites in ACCESS showed greater increases in measures of system integration than comparison sites (Morrissey et al. 2002), clients at these sites did not show greater improvement in housing or symptom outcomes (Rosenheck et al. 2002). However, correlational findings from the ACCESS evaluation showed that sites that had more integrated service systems, regardless of whether they were intervention or comparison sites, had superior 12 month housing outcomes at (Rosenheck et al. 1998) but that this relationship may have been mediated by the level of community social capital, an indicator of overall civic culture (Putnam 1994; 2000; Rosenheck et al. 2001) rather than by specific integration interventions (Rosenheck et al 2002).

In 2002 the federal government made a major commitment to address the problem of chronic homelessness (Sullivan 2002), which was defined in the CICH Notice of Funding Availability (NOFA) as, "an unaccompanied homeless individual with a disabling condition who has either been continuously homeless for a year or more OR has had at least four (4) episodes of homelessness in the past three (3) years." It was expected that by focusing on housing and service needs of this target population it would be possible to disrupt repetitive cycles of recurrent homelessness and free resources to meet the needs of the vast majority of people experiencing homelessness who, with limited assistance, can often exit homelessness relatively quickly (Interagency Council on Homelessness 2003).

A major service demonstration project that emerged from this effort is the Collaborative Initiative to Help End Chronic Homelessness (CICH), a $55 million in federal dollars effort ($35 million funding in 2003, with $20 million added in subsequent years) jointly funded by the Department of Housing and Urban Development, the Department of Veterans Affairs, and the Department of Health and Human Services. After a competitive Request for Applications, CICH awards were made to 11 jurisdictions to provide comprehensive assistance to chronically homeless persons, and to help them move from the streets and emergency shelters into stable housing.

CICH is the first national evaluation of client outcomes among chronically homeless individuals targeted to receive comprehensive housing and support services through a collaboration of HUD, HHS, & VA. Neither the Robert Wood Johnson Foundation Program on Chronic Mental Illness nor the ACCESS program targeted this particularly vulnerable and costly subgroup. Furthermore, no previous initiative focused on collaboration among the three federal agencies primarily responsible for assisting and providing care for homeless persons or emphasized both system integration and implementation of evidence-based practices.

In this report we present system-level observational evaluation data on the CICH program. We describe changes in performance measures over time, by agency type (i.e., lead, housing service provider, veteran service provider, etc) and across sites, as well as examine interrelationships among salient measures. CICH focused on improving outcomes for chronically homeless people by making funding available to provide five core services at each site:  (1) permanent supportive housing, (2) mental health treatment, (3) substance abuse treatment, (4) primary health care, and (5) veteran health services. Services were to be provided through a local network of agencies, coordinated by a local "lead" agency that would oversee distribution of funds; facilitate joint planning and coordination across agencies; promote the use of evidence-based practices; and foster development of homeless management information systems. This evaluation was not designed to evaluate the causal impact of specific interventions but to present descriptive data on service system activities and characteristics, clinical service delivery, and client outcomes.

The CICH program represents an extension of efforts to integrate services for people experiencing homelessness fostered for many years by the Department of Housing and Urban Development's (HUD) Continuum of Care (COC) initiative. Since 1996, through its competitive application process for supportive housing programs, HUD has promoted the development of networks of agencies that together constitute a Continuum of Care. These networks are organized around the centralized distribution and coordination of federal funding to provide a comprehensive array of services for homeless Americans (Burt et al 2002). In addition to encouraging interagency collaboration, CICH took note of but did not mandate the development of "Housing First" service models, in which people experiencing homelessness are moved into permanent housing as quickly as possible, and once housed, are provided long-term support to facilitate access to services thereby preventing a return to homelessness (Tsemberis et al. 2000, 2004; Kowal 2006).

This report examines four questions reflecting central objectives of the CICH initiative at the service system level. First, to what degree is CICH associated with implementation of practices that encourage system integration; with improvements in coordination of service delivery and planning among participating agencies over time; and with increased trust and respect between providers? Second, was the initiative associated with changes in the type of housing provided at CICH sites, with the implementation of homeless information management systems, or with the availability of evidence-based mental health practices? Third, did some sites and some types of agencies show greater change in measures of system-wide performance than others? Lastly, to complement the focus on organizational integration we examine whether relationships specifically characterized by exchanges of funds are associated with greater levels of inter-agency integration, collaboration and trust, both cross-sectionally and over time.

While this report thus focuses on an examination of system-level activities in CICH, another report, "HUD/HHS/VA Collaborative Initiative to Help End Chronic Homelessness National Performance Outcomes Assessment Client Outcomes Report," will present data on service delivery and client outcomes in the CICH program and a third report will address the relationship of client outcomes and system integration.

Methods

Surveys and Data Collection

The CICH program began between March and August 2004 at 11 sites: Los Angeles, Martinez (Contra Costa), and San Francisco, California; Philadelphia, Pennsylvania; New York, New York; Denver, Colorado; Chicago, Illinois; Columbus, Ohio; Chattanooga, Tennessee; Fort Lauderdale, Florida; and Portland, Oregon (See Table 1 for a brief description of each site) and are scheduled to run for a three year period after start up. Sites varied in the number of chronically homeless that they would serve, from 50 to 100 people.

Table 1:
Site Overviews

Site	Description
Broward County, Florida ~ HHOPE Program	The HHOPE program, a collaboration of the Broward County Human Services Department, Homeless Initiative Partnership Administration, provides scattered site housing and supportive services to severely and persistently mentally ill and chronically homeless individuals through Shelter Plus Care. The project is implementing a Housing First approach and using a modified ACT team.  Their goal is to serve 80 individuals over the life of the project.
Chattanooga, Tennessee ~ The Collaborative Initiative	Chattanooga’s Collaborative Initiative, coordinated by the Fortwood Center, serves chronically homeless individuals in scattered site housing.  The Initiative is implementing the Housing First approach and an ACT team to provide wrap-around services for clients in housing.  The goal of the Initiative is to serve 50 individuals over three years, in housing subsidized through Shelter Plus Care.
Chicago, Illinois ~ ACT Resources for Chronically Homeless (ARCH)	Led by the Chicago Department of Human Services, ARCH targets chronically homeless individuals with mental health, substance abuse, and/or co-occurring disorders.  They are using Shelter Plus Care vouchers to secure 59 tenant-based permanent housing units. The units are both scattered site and clustered.  ARCH uses a Housing First approach and an ACT team. Their goal is to bring about significant expansion of permanent supportive housing, coordination and maximization of mainstream resources, and expansion of evidence-based service strategies to meet the complex needs of chronically homeless people.
Columbus, Ohio ~ Rebuilding Lives PACT Team Initiative (RLPTI)	RLPTI is led by Southeast, which contracts project management to the Community Shelter Board.  The project serves chronically homeless individuals with severe mental disabilities or co-occurring substance abuse and mental illness.  For this initiative, they have five clustered site housing units through a Supportive Housing Program grant and use a Housing First approach. They use a PACT model and have incorporated several evidence-based practices.  One of RLPTI’s main goals is to increase the behavioral healthcare system in Franklin County, particularly by increasing Southeast’s capacity and treatment slots. In addition, they plan to increase income supports and entitlements for the chronically homeless. The goal of the RLPTI is to house and serve 108 individuals.
Contra Costa, California ~ Project Coming Home (PCH)	Led by the Contra Costa Office of Homeless Programs, PCH serves chronically homeless individuals using a Health, Housing, and Integrated Services Network (HHISN).  Through Shelter Plus Care, PCH uses a housing first, scattered site model facilitated through partnerships with the housing authority and Shelter Inc.  The goals of PCH include: increasing the effectiveness of integrated systems of care by providing comprehensive services and treatment, linked to housing; increasing the use of mainstream resources that pay for services and treatment; and supporting the development of infrastructures that sustain housing, services treatment, and inter-organizational partnerships beyond the federal initiative.  Over a five-year period, they expect to contact 5,250 chronically homeless individuals and house 155 individuals.
Denver, Colorado ~ Denver Housing First Collaborative (DHFC)	DHFC, a Shelter Plus Care grant is a collaboration of agencies led by Colorado Coalition for the Homeless.  It seeks to provide coordinated housing and treatment to chronically homeless individuals with disabilities, substance abuse, severe and persistent mental illness, co-occurring disorders, and/or chronic physical illness.  DHFC uses a Housing First approach and an ACT team.  Housing is both scattered site and clustered.  DHFC aims to serve 100 clients in year one.
Los Angeles, California ~ Skid Row Collaborative	The Collaborative, which has a Shelter Plus Care Grant, is coordinated by the Skid Row Housing Trust, and seeks to serve chronically homeless and disabled persons.  It uses the Health, Housing, and Integrated Services Network (HHISN) model.  To reach their goal of assisting clients into permanent housing, the Collaborative is expanding mental health and co-occurring treatment services by adding a team of case manager specialists in mental health and substance abuse and peer advocates to provide outreach, engagement, support and recovery services/ treatment, and case management.  They have a goal of housing and serving 62 individuals. The project has already had contact with 140 homeless individuals in its first year.
New York, New York ~ In Homes Now (IN)/Project Renewal	IN, a Supportive Housing grant, is coordinated by Project Renewal serves chronically homeless individuals who are active substance abusers in New York City.  IN uses a Housing First approach and an Intensive Integrated Service Team to supplement existing programs (Continuum of Care and Pathways to Housing) for which active substance users are not eligible.  The project’s goal is to house and provide comprehensive services for 40 individuals from the target population in scattered-site SRO apartments located in Manhattan and the Bronx.
Philadelphia, Pennsylvania ~ Home First	With the City of Philadelphia as the lead agency, Home First serves homeless individuals who have serious mental illness and/or co-occurring disorders and who have among the highest number of documented days in the city’s emergency shelter and residential behavioral health system. They use a Housing First approach and an ACT team.  The project intends to serve approximately 85 chronically homeless individuals over the life of the Supportive Housing Program project.
Portland, Oregon ~ The Community Engagement Program (CEP III)/ Central City Concern	CEP III, which has a Shelter Plus Care grant, is coordinated by Central City Concern.  The project focuses on the “hardest to serve” of Portland’s chronically homeless population – those with a significant disability (i.e., physical health, mental health, and/or substance abuse issues) and/or co-occurring disorders.  Based on ACT and the Housing First approach, clients are housed in scattered site, clustered, or Shelter Plus Care units.  The project’s main goal is to demonstrate an effective model in reducing chronic homelessness for people with co-occurring disorders. CEP III seeks to serve 100 clients in the first year and 150 over the life of the project.
San Francisco, California ~ Direct Access to Housing (DAH)	The San Francisco Department of Public Health is the lead agency for the Direct Access to Housing initiative, which has a Supportive Housing Program grant.  They are creating 70 units of permanent supportive housing through an expansion of their DAH program at the Empress Hotel.  DAH serves chronically homeless individuals with disabilities, using a supportive housing model.

Each site identified core agencies to participate in an annual key informant survey involving representatives of the lead agency and partnering agencies that provided housing assistance, mental health care, substance abuse services, primary care, and the local Veterans Health Administration facility. A "network definition survey" was administered at the start of the initiative (in November/December 2003) to identify participating agencies and key informants at each site. Key informants were those identified by program leaders as the most knowledgeable about the activities of each agency at each site.

A second more extensive "network participation survey" was then administered in three waves, between November 2003 to March 2006, to the key informants identified at each of the participating agencies. The first survey wave occurred before CICH began (from November 2003 to March 2004). The second and third survey waves occurred at the end of the first and second years of operation (from November 2004 to February 2005 and from January to March 2006, respectively). Surveys were sent to all key informants prior to the interviews, which were conducted over the telephone by national evaluation staff at the Northeast Program Evaluation Center. In some sites one agency provided more than one key service (e.g., the mental health agency also provided substance abuse service) and only one survey was conducted for that agency. At other sites, in contrast, more than one agency provided a particular service and key informants at both agencies were interviewed. Thus, the number of agencies at each site ranged from five to nine. Furthermore, at some agencies more than one key informant was identified and interviews were jointly held.

Measures

Five dimensions of CICH service systems were assessed: 1) system connectedness and integration (Coccozza et al., 2000; Interagency Council on Homelessness 2003; Morrissey et al., 2002); 2) emphasis on providing permanent supported housing services (Interagency Council on Homelessness 2003); 3) development of homeless services management information systems (Magnobosco-Bower JL 2001); 4) use of evidence-based mental health practices (DHHS 1999; New Freedom Commission on Mental Health 2003; SAMHSA, 2003); and 5) the existence of interagency fiscal relationships. Because each measurement domain could be addressed in several ways, multiple measures were used in each domain.

Most of the measures were based on items in which each key informant characterized the overall service network at their site from the perspective of their agency. However, elaborating on methods developed by Morrissey et al. (2002), 16 questions asked key informants about the relationship of their agency to each of the other participating agencies in their local network (see items 32-45 in Appendix A). While respondents answered questions about the performance of their network as a whole only once, they gave four to nine responses (depending on the number of agencies in the core CICH network) to questions concerning their dyadic relationship with each other agency. Creation of scales using these 16 items involved multiple steps. First, we conducted a separate exploratory factor analysis of the 16 items. Factor analysis indicated that data could be summarized in four factors that explained 74% of the variation in the 16 measures (see Table 2 for factor loadings for individual items). One of the four factors represented joint planning and cooperation with other agencies, and a second addressed trust and respect between agencies. For these two factors the relevant items were averaged to create scales characterizing each dyadic relationship. The other two factors addressed interagency funding flows: and were used to create a measure indicating whether a fiscal relationship existed between each pair of agencies.

Table 2:
Four-Factor Solution to a Factor Analysis of 15 Dyadic Integration/Coordination
Items from the Network Participation Survey

Burden Item	Factor Loading
Factor 1: Joint Planning and Cooperation
Send clients to or receive homeless clients from this agency	.831
Co-locate staff and/or services in the same location with this agency	.594
Cooperate in serving clients	.912
Communicate clearly with this agency in serving clients	.894
Agree on goals with this agency for  serving clients	.780
Cooperate with this agency in planning	.526
Communicate clearly with this agency in planning	.558
Send information to or receive information from this agency for coordination, control, planning, or evaluation	.539
Have written documents (e.g., MOUs) or flow charts specifying working relationship with this agency	.452
Factor 2: Trust and Respect
Trust this agency	.903
Respect this agency	.905
Factor 3: Send Funds to or Influence Flow of Funds to Other Agency
Send funds to this agency	.920
Influence flow of funds to agency	.889
Factor 4: Receive Funds From or Influences Flow of Funds to Your Agency
Receive funds from this agency	.910
To what extent does this agency influence the flow of funds to you	.906

In the sections below we describe specific measures in each of the five domains.

1) System connectedness and integration: Six measures were used to assess connectedness and integration, four of which were based on key informant ratings of the overall CICH network and two that were based on dyadic (i.e. interagency) relationship scales, as described above.

The first system-level measure assessed the extent to which each network implemented specific interventions that were designed to result in greater system integration (Morrissey 2002). This measure is the average implementation rating given to 20 potential interventions, 12 of which were developed to evaluate the implementation of integration strategies in the ACCESS program (Cocozza et al. 2000; Morrissey et al. 2002) and the remainder of which were developed specifically for the CICH initiative. Each item represented the degree to which each of the 20 strategies was implemented, on a score from zero to three with zero representing "none" or no effort and three representing "a lot". Examples of these strategies are presence of: a) a system integration coordinator position, b) an interagency coordinator body, c) a formal strategic plan, d) co-location of services, e) cross-training, or f) client tracking systems (see Appendix A for a full list of these strategies — items 46-65) (Cronbach's coefficient alpha=.88; Range .2 - 3).

The second system-level integration measure assessed the extent to which the local homelessness coalition provided material resources, or political or institutional support for the implementation of the CICH initiative. This "coalition participation" measure is the average of five items which addressed how important the coalition had been to the CICH network in providing: 1) support for the development of the initial application; 2) material resources, 3) political and institutional support; 4) help in implementing the initiative; and 5) guidance in the shaping of the goals of the initiative (Cronbach's coefficient alpha=.84 for all five items). Items were scored from zero to three with zero representing "not at all" and three representing "very" or "a lot" (see appendix A for details on the specific items in this scale — 86-90).

The third measure of system connectedness and integration was a global assessment of how well the agencies at a site worked together to solve problems. This scale is the average of four items that focus on how well agencies jointly identified, understood and fixed service delivery problems, as well as how effectively they worked together to address client service needs (Cronbach's coefficient alpha=.83). Here too, each item was scored on a zero to three scale where zero represented "not at all" and three represented "very well" (see Appendix A items 18a-18d).

The fourth system integration scale assessed the total number of different types of agencies that were involved with the CICH network. Key informants at each surveyed agency were asked about agencies that were on the initial grant application for the CICH network and agencies as well as types of local organizations that were not included among the core agencies (e.g., soup kitchens, law enforcement agencies, private businesses, etc.) (Cronbach's coefficient alpha=.75; Range 3 - 12). Items in this scale were dichotomized so that answers of two or three ("somewhat involved" or "very involved") were scored as a one, and other responses were scored as zero (see Appendix A items 11a-11l).

The last two scales were based on items derived from the dyadic measures of the relationships between pairs of agencies and addressed a) joint planning and cooperation and b) trust and respect within each dyadic relationship. First, as suggested by the factor analysis described above, nine items were averaged to constitute a scale representing inter-agency joint planning and coordination. Specific items in this scale addressed: cooperation in serving clients, goal congruence, client referral, cooperative planning, co-location of staff and services, information sharing and communication (see items 32-36 and 40-43 in appendix A for details).

The second dyadic interagency scale, a measure of interagency trust and respect, was constructed as the average of two items addressing the extent to which this relationship was characterized by: 1) trust and 2) respect.

Items in all dyadic scales were scored from zero to three with zero representing "none" and three representing "a lot".

2) Emphasis on providing diverse housing services. Two measures were derived from an exploratory factor analysis of eight items and addressed: a) degree of emphasis in the network, as a whole, on providing various types of housing; and b) changes in the types of housing emphasized by each CICH network (for further details see items 13a-13d and 15a-15d in Appendix A). Factor analysis indicated that these items could be summarized in two scales, one reflecting an emphasis on emergency shelter as well as transitional and unsupported housing and the other reflecting a change in these emphases (see Table 3 for factor loadings for individual items).

Table 3:
Two-Factor Solution to a Factor Analysis of 8 Housing Emphasis Items
from the Network Participation Survey

Burden Item	Factor Loading
Factor 1: Current Housing Emphasis by CICH network
Emergency Shelter	.749
Transitional Housing	.829
Affordable Housing	.645
Factor 2:  Change in Housing Emphasis by CICH Network
Emergency Shelter	-.659
Transitional Housing	-.728
Permanent Supported Housing	.578

3) Development of homeless services management information systems (HMIS). Two measures were created to evaluate the degree to which systems existed at the network level for management of client, service delivery and outcome information. The first measure was the average of three items that documented whether a system was available to generate data on: 1) client characteristics, 2) housing and service delivery, or 3) client outcomes. These three items were scored from zero or one, with zero representing "no" and one representing "yes".

The second measure was a single item which asked if an HMIS specific to the CICH initiative had been implemented. There were five possible answers to this question: 1) neither planned for, nor currently available for use, 2) being planned or developed, 3) in use by some network members, 4) in use by most network members, or 5) in use by all network members. The first answer was scored as zero while the last answer was scored as four (see items 19, 21, 23, and 25 in Appendix A for more details).

4) Evidence-based practices. A single scale was used to measure the degree to which each of the 11 CICH networks was viewed by its core agencies as having implemented evidence-based practices. This scale was based on the 18 practices identified in 2003 by the Substance Abuse and Mental Health Services Administration (SAMHSA) to be solidly evidence-based in a report entitled "An Overview of Mental Health and Substance Abuse Services and Systems Coordination Strategies" (SAMHSA 2003). Examples of these practices are: a) multidisciplinary treatment teams or intensive case management (e.g. Assertive Community Treatment [ACT]); b) self-help programs; c) the housing-first model of supportive housing; d) opiate substitution; and e) family psycho-education (for further details see items 66 to 82 in Appendix A). Items for each of the 18 evidence-based practices measured the extent to which these practices were implemented at each site and were scored from zero to three, with zero representing "none" or no effort and three representing "a lot". The values for the 18 items, one for each evidence-based practice, were averaged for each survey respondent to create this scale (Cronbach's coefficient alpha= .86; Range .11 - 3).

5) Resource flows and influence over resource flows. Finally, a scale was constructed to measure whether an active fiscal relationship existed between each pair of agencies. This measure was based on four items based on the factor analysis described above. First, two intermediate scales were created that characterized the transfer of funds or influence over the transfer of funds between agencies. The first of these dyadic scales (sending or influencing funds sent), was based on responses to two questions concerning the extent to which each agency: a) directly sent funds to each other agency or b) influenced the flow of funds to other agencies from third parties. The second of these scales (receiving funds or influencing receipt of funds) was based on answers to a related pair of questions concerning the extent to which each agency: a) received funds directly from each other agency or b) influenced the receipt of funds from each other agency. The answers to these individual items were scored from zero to three with zero representing "no funds exchanged" and three representing "a lot of funds exchanged". These two intermediate scales were constructed by averaging each pair of items. In the second step, a final, dichotomous measure was created that had a value of one if either of the two intermediate scales had a value of one or greater (i.e., a rating of at least "a little" on either of the funds transfer measures). Otherwise the scale had a value of zero (no funds transfer). For every pair of agencies this scale indicates whether there existed any resource transfer, or influence over resource transfer from a third party between the two agencies in the dyad.

Analyses

Separate analyses were conducted to address each of the central study questions.

System Integration: The first question concerned the degree to which the CICH project was associated with the implementation of interventions designed to increase system integration, and/or with resultant increases in system coordination, or in trust and respect among agencies at each site. To address this question. a series of analyses were conducted in which the dependent variables were the six system connectedness and integration measurements discussed previously, and the independent variable was a categorical variable representing the year of the project (time). This variable had a value of one to three, with one indicating measurement conducted before the project started and values of two or three indicating measurements that occurred in subsequent years.

In these analyses random effects were modeled using an unstructured covariance matrix, thereby adjusting standard errors for the correlated nature of the data in these models (i.e., for the potential correlation of observations from the same agency for different years). This technique is often referred to as hierarchical linear modeling (HLM)⁽³⁾ (38).

The second part of the first evaluation question was whether the implementation of practices designed to facilitate system integration was associated with observed increases in system coordination as well as in trust and respect among providers at each site. To investigate this issue we examined the correlation of measures reflecting the implementation of integration strategies, goals and activities with the measures of system coordination and the measure of trust and respect. Correlations at the agency level were done using the PROC CORR procedure of the SAS ® software system.

Other System Changes: Analysis of data to answer the second study question also focused on system change, in this case, in the types of housing provided, in the degree to which information systems were implemented, and in the extent to which evidence-based mental health practices were deployed. The analyses of these data were similar to those described for measures of integration. Separate HLM models were examined in which the dependent variables were the measures of system performance and the independent variable represented the year of the study.

Site and Agency Change: The third system-level question concerned variation across sites and agency types in integration efforts and outcomes, and in other system-level performance measures.

In the analysis of the degree to which the 12 system performance measures changed to different degrees across sites, analysis of covariance was used to test the relationship between a categorical variable with 11 levels representing the site of the responding agency, a variable representing time, and the variable of primary interest, a term that represented the interaction of site and time. These analyses thus examined whether there were significantly different degrees of change over time between sites, on each dependent measure. For those dependent measures in which the interaction term was significantly related to the dependent measure (with an alpha level set to less than 0.15 because of the small number of observations), we further examined each measure over time to identify specific sites at which there was statistically significant change over time. The PROC GLM procedure of SAS ® was used to conduct these analyses.

The same analytic approach was used to examine variation across agency types over time on these measures. In these analyses a measure that represented the type of respondent agency was substituted for the measure of site. This measure is a categorical variable that represents each of the seven types of agencies, i.e., the lead agency; the local Veterans Health Administration facility; or an agency that provided housing assistance, mental health care, substance abuse services, primary care, or another service.

Fiscal Integration. The final system level question addressed the association between the presence of a funding relationship between each pair of agencies and the degree of joint planning and cooperation, as well as trust and respect. In this analysis, we first used HLM to examine two models in which the independent measure was the dyadic indicator of the existence of a fiscal relationship. Random effects were modeled for site, agency, and time. An additional set of models addressed the interaction of fiscal relationship and time. These models explored whether there were differences in the amount of change in the dependent measures between dyads with and without a fiscal relationship.

Results

Sample Characteristics

Analyses were based on two types of measures, as described above, one that used each participating agency at each of the 11 CICH sites as the unit of analysis, and the other that used dyadic relationships among the agencies as the unit of analysis. An average of 6.7 agencies (standard deviation=1.66) were surveyed at each site in each wave. Data were available for 80 agencies surveyed in wave one; 72 in wave two; and 70 in wave three.⁽⁴⁾

Data were available for a substantially greater number of dyadic relationships — 528 in wave one; 474 in wave 2; and 462 in wave 3. Data were thus available for an average of 44.4 dyadic relationships (standard deviation=19.8) per site in each survey year. Table 4 provides descriptive information on each system level measure and Table 5 presents a summary of bivariate correlations among the 12 measures.

Table 4:
Descriptive Characteristics: System-Level Measures Across All Time Points

Variable Groups	Variables	N	Mean Scores	SD	Range
System Connectedness And Integration	Integrative Practices	215	2.08	.52	0-3
	Coalition Involvement	187	2.25	.72	0-3
	Working Together	210	2.66	.51	0-3
	Involvement of Local Organizations: Number	211	7.65	2.42	0-12
	Dyadic Joint Service Planning and Coordination	1,366	1.82	.87	0-3
	Dyadic Trust and Respect	1,369	2.64	.60	0-3
Emphasis On Providing Housing Services And On The Associated Goal Of Ending Homelessness	Emphasis on Transitional, Emergency, and Affordable Housing as Contrasted with Permanent Supported Housing	206	2.23	.72	0-3
	Change in Emphasis from Emergency Shelter and Transitional Housing to Permanent Supported and Non-supported Housing	208	2.12	.52	1-3
Development Of Homeless Services Management Information System	Client and Services Information Available	208	.84	.29	0-1
	CICH Management Information System Exists	199	1.51	1.01	0-4
Use of Evidence-Based mental health Practices	Use of  Evidence Based Practices	208	2.30	.44	.11-3
Funds Transfer and Influence	Fiscal Relationship Exists	1,355	.31	.46	0-1

System Integration

Results for both the implementation of system integration practices, and the integration outcome measures showed positive trends (Table 6). Over the study period there was a significant increase of 15% in the measure of implementation of practices designed to encourage system integration. While there was no significant changes from wave one to wave three in the level of involvement of the local homeless coalition, or in how well the agencies at each site globally rated the way they worked together, this mostly likely reflects a ceiling effect since participating sites had achieved high baseline scores (2.4-2.6 out of a possible 3.0) prior to program implementation, i.e., during and even prior to the development of their CICH proposals.

There were significant increases in two other key measures of system integration (Table 6). Strong results were observed on the measure of joint planning and coordination, which increased by 25% over the study period and showed highly significant change (p<.0001) (see Figure 1). The dyadic measure of trust and respect also increased by a statistically significant 3.5%. The smaller magnitude of change on this measure similarly reflects a high baseline level (2.6 out of a possible 3.0) that left little room for improvement on the underlying metric.

Figure 1.
System Integration over Time

Analyses of correlations between the implementation measure and the level of system integration actually achieved showed that greater implementation of integration practices was highly and significantly associated with greater system integration and better system performance on virtually all measures (see first column of Table 5).

Other System Changes

Table 7 presents data that address the second study question; i.e., was the implementation of the CICH initiative associated with changes in the type of housing provided at CICH sites, in the implementation of homeless management information systems (HMIS), or in the use of evidence-based mental health practices?

Table 7:
Comparisons Over Time of Other System Characteristics

Variable Groups	Variables	N	Mean Scores			Time Comparison
			Wave 1	Wave 2	Wave 3	Percent Change (Wave 1 to Wave 3)	P*	Pairwise Comparison**
Emphasis On Providing Housing Services And On The Associated Goal Of Ending Homelessness	Emphasis on Transitional, Emergency, and Affordable Housing as Contrasted with Permanent Supported Housing (Factor Score)	206	2.33	2.21	2.15	-7.7%	.32	None
	Change in Emphasis from Emergency Shelter and Transitional Housing to Permanent Supported Housing (Factor Score)	208	2.12	2.07	2.18	2.8%	.47	None
Development Of Homeless Services Management Information System	Client and Services Information Available	208	.75	.86	.90	20.0%	.0009	W3>W1, W2>W1
	CICH Management Information System Exists	199	1.23	1.47	1.90	54.5%	.0006	W3>W1, W3>W2
Use of Evidence-Based mental health Practices	Use of  Evidence Based Practices (Mean Rating)	204	2.12	2.35	2.41	13.7%	.0002	W3>W1, W2>W1
Funds Transfer and Influence	Fiscal Relationship Exists	1,355	.29	.32	.33	13.8%	.52	None

*Proc mixed used in which autocorrelation controlled for with respect to site for those measures based on key informant observations (site and ID for those based on relationship characteristics)

 **Only those comparisons with significance in which p<.05 reported

There was no significant change in the degree to which agencies at CICH sites emphasized the provision of emergency, transitional, and affordable housing without support, as contrasted with permanent supported housing, nor was there significant change in the assessment of change in types of housing emphasized at CICH sites. The lack of change in housing emphasis may reflect ceiling effects since many sites selected for CICH were already committed to developing permanent supported housing.

There was, however, a significant increase of 20% in the reported ability of CICH agencies to obtain information about clients served and services delivered to them by the CICH network. There was also a significant 54.5% increase in the measure of implementation of a homeless management information system.

The 13.7% increase in the measure of the use of evidence-based mental health practices progressed monotonically from wave one to wave three and was also highly statistically significant (p=.0002).

Change by Site and Agency Type

There were few differences among sites, or among types of agencies, in the magnitude of changes in system-wide performance measures.⁽⁵⁾  Significant variation in change by site was observed on the dyadic measure of joint planning and coordination (see Table 8 columns 3-5).

Table 8:
Comparison of Measures by Site and Agency Type

Variable Groups	Variables	N	Site Comparison (All Time Points)		Agency Type Comparison (All Time Points)
			Interaction Effect — Site and Time	Sites that Showed Significant Change Over Time	Interaction Effect — Agency Type and Time	Types of Agencies that Showed Significant Change Over Time
System Connectedness And Integration	Integrative Practices	215	(p=.49)	NA	(p=.85)	NA
	Coalition Involvement	187	(p=.91)	NA	(p=.57)	NA
	Working Together	210	(p=.70)	NA	(p=.83)	NA
	Involvement of Local Organizations: Number	211	(p=.96)	NA	(p=.91)	NA
	Dyadic Joint Service Planning and Coordination	1,366	(p=.02)	Los Angles (p=.0001) Columbus (p=.0001)  Fort Lauderdale (p=.0007) San Francisco (p=.0015) Portland (p=.0195)	(p=.06)	Lead (p=.0045)  Housing (p=.0087)  Mental Health (p=.0017)  VA (p=.0001)  Other (p=.0005)
	Dyadic Trust and Respect	1,369	(p=.28)	NA	(p=.11)	Lead (p=.0046)
Emphasis On Providing Housing Services And On The Associated Goal Of Ending Homelessness	Emphasis on Transitional, Emergency, and Affordable Housing as Contrasted with Permanent Supported Housing	206	(p=.72)	NA	(p=.86)	NA
	Change in Emphasis from Emergency Shelter and Transitional Housing to Permanent Supported Housing	208	(.97)	NA	(p.=62)	NA
Development Of Homeless Services Management Information System	Client and Services Information Available	208	(p=.73)	NA	(p=.81)	NA
	CICH Management Information System Exists	199	(p=.60)	NA	(p=.17)	NA
Use of Evidence-Based mental health Practices	Use of  Evidence Based Practices	208	(p=.49)	NA	(p=.74)	NA
Funds Transfer and Influence	Fiscal Relationship Exists	1,355	(p=.74)	NA	(p=.25)	NA

Analysis of the site changes over time showed five of the eleven sites with significant increases in the level of joint planning and coordination, in contrast to the other six sites that did not show significant change (Table 8). Closer examination of the data indicated that although on average these six sites started at higher level of joint planning and coordination than the other five sites, they all scored less than two (with three being the highest possible score).

Examination of differences in change across types of agencies revealed significant differences in changes on both the measure of joint planning and coordination and on the measure of trust and respect. Five of the seven agency types showed significant change on the joint planning and coordination measure (Table 8 second panel). The lead agency alone showed a significant increase in trust and respect in comparison to the other agencies.

Fiscal Relationship Comparisons

There was no significant increase or decrease in the prevalence of fiscal relationships⁽⁶⁾ among agencies participating in CICH (see last row of Table 9) suggesting that participating in CICH was not associated with the formation of these types of relationships. Table 7 presents comparisons between pairs of agencies (i.e., dyads) with and without a fiscal relationship. Cross sectionally, agencies in dyads characterized by fiscal relationships had substantially higher ratings on the measure of joint planning and coordination (55.5% higher) and on the measure of trust and respect (8.6% higher) than those dyads without a fiscal relationship. However, there were no significant differences in the magnitude of change over the study period between pairs of agencies with and without fiscal relationships. Increases in system integration observed over time cannot therefore be attributed to the development of stronger fiscal relationships.

Table 9:
Funds Transfer Relationship and Measures of Integration

Variables	N	Mean Scores			Presence of Fiscal Relationship	Interaction Effect — Fiscal Relationship and Time
		No Relationship	Relationship Exists	Percentage Difference
Dyadic Joint Service Planning and Coordination	1,326	1.55	2.41	55.5%	(p<.0001)	(p=.10)
Dyadic Trust and Respect	1,322	2.57	2.79	8.6%	(p<.0001)	(p=.95)

Discussion

This report highlights several positive trends in the characteristics and activities of CICH networks over the course of this initiative. The most notable trend was the significant increase over the study period in the implementation of practices that encourage system integration, as well as in levels of system integration themselves, particularly on the measure of joint service planning and coordination. We also found that the implementation of practices intended to encourage system integration was significantly and positively correlated with measured levels of integration. Future reports will evaluate the effects of system integration on service delivery and client outcomes.

It is further encouraging that there was a significant increase in the availability of information on client and service delivery and in the implementation of homeless management information systems as well as in the use of evidence-based mental health practices. Each of these findings reflects movement towards meeting CICH program goals.

There were no significant changes in ratings of the type of housing provided or in the prevalence of dyadic fiscal relationships. The CICH sites were selected as the best candidates out of a field of over 100 applicants and as a result of this selection process these sites appear to have been focused on the provision of permanent supported housing even before the initiative began.

Significant variation among sites or agency types in the amount of change they experienced over the study period was limited to two dyadic measurements of joint planning and coordination and trust and respect. These modest cross-site differences partly reflect lack of statistical power due to the small number of cases as well as to ceiling effects on these measures since these sites were selected for their demonstrated capacity to function as a coordinated network of agencies.

While a major emphasis in CICH was placed on encouraging system integration it was also of interest that agencies with ongoing fiscal relationships had significantly higher levels of joint planning and coordination as well as trust and respect, although these relationships do not explain the changes in integration measures observed over the course of the initiative.

As with previous evaluations of initiatives intended to increase system integration (i.e., the Program on Chronic Mental and the Access to Community Care and Effective Services and Supports program) we found that the CICH initiative was associated with increasing levels of coordination and communication as well as trust and respect among service agencies. Given appropriate motivation, technical support, and resources, participating agencies achieved increased levels of system integration and furthered the implementation of both homeless management information systems and evidence-based practices.

This preliminary evaluation report does not address the issue of whether the increases in system integration, or in the implementation of either information systems or evidence-based practices, were associated with improved client outcomes. Although substantial data will be available on client outcomes, the small number of sites and the high level of integration at the beginning of the project may limit our eventual ability to demonstrate such relationships.

Limitations and Conclusion

Several limitations of this evaluation need mention. Most importantly, all data reported here were based on interviews with a small number of key informants. We did not have access to objective measures of site performance, a weakness that is broadly inherent in research on service systems due to their complexity, the large number of individuals and organizations that comprise them, and the many exogenous environmental factors that influence system operation (Rosenheck et al. 2001).

Secondly, this study did not use an experimental evaluation design through which sites would have been randomly assigned to a treatment or a control group exposed to different interventions, nor did we use a quasi-experimental study design, such as matched site comparisons. Although the pre- and post-implementation data for this evaluation are suggestive, other factors including reporting biases may have been responsible for the measured system changes.

While these limitations prevent us from concluding definitively that the CICH initiative caused the observed system changes at these 11 evaluation sites, the data presented are clearly consistent with such an inference, and suggest that site level initiatives in the CICH program successfully accomplished the program objectives.

References

Burt, Martha, Dave Pollack, Abby Sosland, Kelly Mikelson, Elizabeth Drapa, Kristy Greenwalt, and Patrick Sharkey. (May 2002) Evaluations of Continuums of Care for Homeless People. Washington D.C.: U.S. Department of Housing and Urban Development Office of Policy Development and Research.

Cocozza, J.J; Steadman, H.J; Dennis, D.L; Blasinsky, M; Randolph, F.L; Johnsen, M; Goldman, H. (2000). Successful systems integration strategies: the ACCESS program for persons who are homeless and mentally ill. Administration and Policy in Mental Health 27(6): 395-407

Department of Health and Human Services (DHHS). Mental Health A Report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institute of Health, National Institute of Mental Health, 1999.

Drake RE, Goldman HH, Leff HS, et al. "Implementing Evidence-Based Practices in Routine Mental Health Service Settings" Psychiatric Services 52 (2001): 179-182.

Foster-Fishman, P.G., Salem, D.A., Allen, D.A., & Fahrbach, K. (2001). Facilitating interorganizational exchanges: The contributions of interorganizational alliances. American Journal of Community Psychology, 29(6), 875-905.

Goldman, Ganju, Drake, et al. "Policy implications for implementing evidence-based practices." Psychiatric Services 52(1) Dec 2001 52(12).

Guyatt, G., D Cook, B Haynes. Evidence based medicine has come a long way. BMJ. 2004; 329 (7473); 990-1 30 Ocbert.

McKinney, M M J P Morrissey, and A D Kaluzny. Interorganizational exchanges as performance markers in a community cancer network. Health Serv Res. 1993 October; 28(4): 459-478.

Interagency Council on Homelessness. January 27 2003 Federal Register http://www.hud.gov/offices/cpd/homeless/apply/2002nofa/nofa.doc [Docket No. FR-4805-N-01] Accessed June 1, 2006

Isett, Kimberley; Morrissey, Joseph. Assessing Delayed Effects of a Multi-Site System Intervention for Homeless Persons with Serious Mental Illness. Administration and Policy in Mental Health, Volume 33, Number 1, January 2006, pp. 115-121(7)

Kowal, Jessica. (June 30, 2006) "Homeless alcoholics receive a permanent place to live, and drink." New York Times. Accessed on Internet.

Lehman AF, Postrado LT, Roth D, McNary SW, Goldman HH. "Continuity of Care and Client Outcomes in the Robert Wood Johnson Foundation Program on Chronic Mental Illness." Milbank Quarterly, 1994;72(1):105-22.

Magnobosco-Bower JL. An evaluation of state public mental health system performance for adult persons with serious mental illness: Effects of state political culture and state mental health planning & implementation characteristics on state public mental health system comprehensiveness. Doctoral thesis. Columbia University, 2001

Morrissey, Joseph P., Michael Calloway, et al. Local Mental Health Authorities and Service System Change: Evidence from the Robert Wood Johnson Foundation Program on Chronic Mental Illness. Milbank Quarterly Volume 72 Number 1, 1994

Morrissey JP, Calloway MO, Thakur N, Cocozza J, Steadman HJ, Dennis D; ACCESS National Evaluation Team. Integration of service systems for homeless persons with serious mental illness through the ACCESS program. Access to Community Care and Effective Services and Supports. Psychiatr Serv. 2002 Aug; 53(8):949-57.

New Freedom Commission on Mental Health, Achieving the Promise: Transforming Mental Health Care in America. Final Report. DHHS Pub. No. SMA-03-3832. Rockville, MD 2003.

Provan Keith G. Brinton Milward. A Preliminary Theory of Interorganizational Network Effectiveness: A Comparative Study of Four Community Mental Health Systems. Administrative Science Quarterly, Vol. 40, 1995. 1-33.

Putnam RD, Leonardi R, Nanetti RY: Making Democracy Work: Civic Traditions in Modern Italy. Princeton University Press, New Jersey, 1994

Putnam RD: Bowling Alone: The Collapse and Revival of American Community. New York, Simon and Schuster, 2000.

Randolph, Frances, Margaret Blasinsky, M.A., Joseph P. Morrissey, Ph.D., Robert A. Rosenheck, M.D., Joseph Cocozza, Ph.D., Howard H. Goldman, M.D., Ph.D. and the ACCESS National Evaluation Team Overview of the ACCESS Program. Psychiatr Serv 53:945-948, August 2002

Rosenheck RA, Morrissey J, Lam J, Calloway M, Johnsen M, Goldman HH, Calsyn R, Teague G, Randolph F, Blasinsky M and Fontana A (1998) Service System Integration, Access to Services and Housing Outcomes in a Program for Homeless Persons with Severe Mental Illness. American Journal of Public Health 1998; 88(11):1610-1615.

Rosenheck RA, Morrissey J, Lam J, Calloway M, Stolar M, Johnsen M, Randolph F, Blasinsky M and Goldman HH. Service delivery and community: Social capital, service systems integration, and outcomes among homeless persons with severe mental illness. Health Services Research 2001;36(4):691-710.

Rosenheck RA, Lam J, Morrissey JP, Calloway M, Marilyn Stolar, Randolph F, Blasinsky M, Johnsen, M, Steadman H, Cocozza J, Dennis DD, Goldman HH. Service systems integration and outcomes for mentally ill homeless persons in the ACCESS program. Access to Community Care and Effective Services and Supports. Evidence from the ACCESS Program. Psychiatric Services. 2002:53(8):958-966

SAMHSA. Best practices: An overview of mental health and substance abuse services and system coordination strategies. URL: http://alt.samhsa.gov/grants/content/2003/Resource%20material.doc (2003). Accessed June 1, 2006

Sullivan, Brian. (July 18, 2002). "News Release: White House Interagency Council announces new strategy to combat chronic homelessness: New plan will better coordinate federal response to homelessness." Interagency Council on Homelessness. Accessed at http://www.hud.gov/news/release.cfm?content=pr02-078.cfm on July 13, 2006.

Torrey, William C M.D., Robert E. Drake, M.D., Ph.D., Lisa Dixon, M.D., M.P.H., Barbara J. Burns, Ph.D., Laurie Flynn, A. John Rush, M.D., Robin E. Clark, Ph.D. and Dale Klatzker, Ph.D. Implementing Evidence-Based Practices for Persons With Severe Mental Illnesses. Psychiatric Services 52:45-50, January 2001

Tsemberis, Sam. and Ronda F. Eisenberg, M.A. Pathways to Housing: Supported Housing for Street-Dwelling Homeless Individuals With Psychiatric Disabilities. Psychiatric Services 51:487-493, April 2000

Tsemberis S, Gulcur L, Nakae M. Housing First, consumer choice, and harm reduction for homeless individuals with a dual diagnosis. Am J Public Health. 2004 Apr; 94(4):651-6.

Appendix A: Network Participation Survey

Agency ID [Entry Field]

Interviewer ID [Entry Field]

Date of Survey [Entry Field]

Agency Name [Entry Field]

Agency Representative [Entry Field]

Phone Number [Entry Field]

Sequence Number 1 = Baseline, 2 = Year 1, 3 = Year 2, 4 = Year 3, 5 = Other (Specify: ___________)

1. For which component(s) of the Chronic Homelessness Initiative is this agency/organization responsible? (Code ‘0’ or ‘1’ for each item below)
   1. Lead agency [0 = No 1 = Yes]
   2. Permanent housing service provider [0 = No 1 = Yes]
   3. Primary health care services provider [0 = No 1 = Yes]
   4. Mental health services provider [0 = No 1 = Yes]
   5. Substance abuse treatment provider [0 = No 1 = Yes]
   6. Veteran service provider (VA or community provider) [0 = No 1 = Yes]
   7. Other service provider (e.g., partnering agency) [0 = No 1 = Yes]
2. SKIP QUESTIONS 2-3 FOR YEARS 1-3
3. SKIP QUESTIONS 2-3 FOR YEARS 1-3
4. About how many agencies/organizations are involved in the CHI network? [Entry Field, Code Don’t Know = ’99’]
5. How often does the CHI network meet, on average?
   1. = More Than Weekly
   2. = Weekly
   3. = Twice/month
   4. = Monthly,
   5. = Bi-monthly
   6. = Quarterly
   7. = Bi-annually
   8. = Annually
   9. = DK
6. Does the CHI network have a formal committee structure? [Entry Field 0 = No, 1 = Yes, 9 = DK (If ‘No’ or ‘DK’, SKIP to Q10)
7. Please specify the names of a few committees.
   1. Name: [Entry Field]
   2. Name: [Entry Field]
   3. Name: [Entry Field]
8. How effective are these committees in accomplishing the goals of the CHI network? [Entry Field, 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very, 9 = DK]
9. Are consumers on these committees? [Entry Field, 0 = No, 1 = Yes, 9 = DK]
10. Sometimes there are critical individuals without whom the network would not be sustainable. Are there key leaders without whom the network
    1. Would not have been established? [Entry Field, 0 = No, 1 = Yes, 9 = DK]
    2. Would not continue to function at its current high level? [Entry Field, 0 = No, 1 = Yes, 9 = DK ]
    3. Please name up to two of these key individuals [Entry Field 1] [Entry Field 2]
11. To what extent has each of the following agencies been actively involved with the CHI network during the past year? (0 = Not at all 1 = A little bit 2 = Somewhat involved 3 = Very involved 9 = DK)
    1. Public housing authority (PHA) [Entry Field]
    2. Social Security Administration (SSA) [Entry Field]
    3. City or county welfare/general assistance office [Entry Field]
    4. Soup kitchens [Entry Field]
    5. Vocational rehabilitation agencies [Entry Field]
    6. Law enforcement [Entry Field]
    7. Faith based organizations [Entry Field]
    8. Private businesses [Entry Field]
    9. Primary health care services provider [Entry Field]
    10. Mental health services provider [Entry Field]
    11. Substance abuse treatment provider [Entry Field]
    12. Veteran service provider [Entry Field]
12. Is there a local governmental agency responsible for the care of homeless persons in your community? [Entry Fireld, 0 = No, 1 = Yes, 9 = DK, (If ‘No’ or ‘DK’, SKIP to Q13)
    • A1. What type of agency is this?
      1. a state agency? [Entry Field, 0 = No, 1 = Yes, 9 = DK]
      2. a county agency? [Entry Field, 0 = No, 1 = Yes, 9 = DK]
      3. a local agency? [Entry Field, 0 = No, 1 = Yes, 9 = DK]
      4. any other? [Entry Field, 0 = No, 1 = Yes, 9 = DK (Specify_)]
    • How involved has this government agency been with the network during the past year? [Entry Field, 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very involved, 9 = DK]
13. To what extent does (do) the provider(s) in the CHI Network considered together now provide each of the following types of housing for homeless people through the use of either CHI funding or non-CHI funding? [Entry Field, 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK]
    1. Emergency shelter [Entry Field]
    2. Transitional housing [Entry Field]
    3. Affordable housing without services [Entry Field]
    4. Permanent supported housing (i.e., housing with significant supportive services) [Entry Field]
14. In recent years has there been a change in emphasis in the type of housing provided by the agencies participating in the CHI network? [Entry Field] 0 = No, 1 = Yes, 9 = DK, (If ‘No’ or ‘DK’, SKIP to Q16)
15. How has the emphasis changed for each of the following types of housing (i.e., housing emphasized less, same, or more)?
    • Emergency shelter [Entry Field] 1 = Less emphasis, 2 = Same, 3 = More emphasis, 9 = DK
    • Transitional housing [Entry Field] 1 = Less emphasis, 2 = Same, 3 = More emphasis, 9 = DK
    • Permanent non-supported housing [Entry Field] 1 = Less emphasis, 2 = Same, 3 = More emphasis, 9 = DK
    • Permanent supported housing [Entry Field] 1 = Less emphasis, 2 = Same, 3 = More emphasis, 9 = DK

    
    GOALS AND FOCUS

16. Which of the following goals are most important for the CHI network? Next most… (Code 1-5, with most important = ‘1’ and least important = ‘5’)
    1. Obtain grant funding from the Chronic Homelessness Initiative [Entry Field]
    2. Obtain grant funding from other sources [Entry Field]
    3. Establish partnerships among homeless service providers to better coordinate care across agencies [Entry Field]
    4. Develop an integrated system of care for persons who are homeless [Entry Field]
    5. Develop systematic plans to end or substantially reduce the prevalence of chronic homelessness in our community [Entry Field]

    
    PLANNING

17. Did the CHI network conduct the following planning activities focused on ending chronic homelessness during the past year or were they involved in such activities as part of a larger coalition?
    1. Developed a mission statement [Entry Field] 0 = No, 1 = Yes, 9 = DK
    2. Identified goals for training staff [Entry Field] 0 = No, 1 = Yes, 9 = DK
    3. Used quantitative methodology to determine the prevalence and/or unmet needs of homeless people [Entry Field] 0 = No, 1 = Yes, 9 = DK
    4. Conducted an inventory of available housing and supportive services for homeless people [Entry Field] 0 = No, 1 = Yes, 9 = DK

    
    MANAGEMENT
     

18. How well does the CHI network work together to…
    1. Identify barriers or bottlenecks in delivery of housing and services to homeless people? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very well, 9 = DK
    2. Understand the reasons for these problems? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very well, 9 = DK
    3. Fix or improve these problems? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very well, 9 = DK
    4. Address individual client service needs? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very well, 9 = DK
19. Do you currently have a way of identifying the characteristics of the people served by the CHI network? [Entry Field] 0 = No, 1 = Yes, 9 = DK, (If ‘No’ or ‘DK’, SKIP to Q21)
20. Does the system provide information on client characteristics…?
    • In real-time, used by clinicians in the delivery of services [Entry Field] 0 = No, 1 = Yes, 9 = DK
    • In the form of aggregated periodic summaries [Entry Field] 0 = No, 1 = Yes, 9 = DK
    • As individual client data [Entry Field] 0 = No, 1 = Yes, 9 = DK
    • Through an inter-agency management information system [Entry Field] 0 = No, 1 = Yes, 9 = DK
21. Do you currently have a way of identifying the housing and services
    delivered by the CHI network? [Entry Field] 0 = No 1 = Yes 9 = DK, (If ‘No’ or ‘DK’, SKIP to Q23)
22. Does the system provide data on housing and services delivered …?
    1. In real-time, used by clinicians in the delivery of services [Entry Field] 0 = No, 1 = Yes, 9 = DK
       In the form of aggregated periodic summaries [Entry Field] 0 = No, 1 = Yes, 9 = DK
       As individual client data [Entry Field] 0 = No, 1 = Yes, 9 = DK
       Through an inter-agency management information system [Entry Field] 0 = No, 1 = Yes, 9 = DK
23. Do you currently have a way of determining the client outcomes of clients served by the CHI network? [Entry Field] 0 = No, 1 = Yes, 9 = DK, (If ‘No’ or ‘DK’, SKIP to Q25)
24. Does the system provide data on client outcomes …?
    1. In real-time, used by clinicians in the delivery of services [Entry Field] 0 = No, 1 = Yes, 9 = DK
    2. In the form of aggregated periodic summaries [Entry Field] 0 = No, 1 = Yes, 9 = DK
    3. As individual client data [Entry Field] 0 = No, 1 = Yes, 9 = DK
    4. Through an inter-agency management information system[Entry Field] 0 = No, 1 = Yes, 9 = DK
25. Is there a CHI network management information system [Entry Field] 0 = Neither planned for nor currently in use, 1 = Being planned for or developed, 2 = Being used by some network members, 3 = Being used by most network members, 4 = Being used by all network members, 5 = Other [Entry Field](If ‘No’ or ‘DK’ or ‘1’, SKIP to Q27), 6 = DK
26. To what extent has the management information system been used for each of the following purposes?
    1. To identify malfunctioning processes of the service delivery system for homeless people [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very effectively, 9 = DK
    2. To understand causes of these malfunctioning processes [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very effectively, 9 = DK
    3. To fix or improve these malfunctioning processes [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very effectively, 9 = DK
    4. To address individual client service needs [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very effectively, 9 = DK

    
    SKIP QUESTIONS 27-31 FOR YEARS 1-3

    
    CONNECTIVENESS AND INTEGRATION

27. During the past year, to what extent has your agency participated in the following activities with each member of the CHI network? Note that all your responses will be confidential. (Code each line below 0-5. The letters above each column refer to the agencies listed in Q27. Read questions 32-45 for each agency, beginning with the first column and ending with the last column. Code rows under your agency column ‘4’.) 0 = None, 1 = A little, 2 = Some, 3 = A lot, 4 = N/a (own agency), 5 = DK
    Agencies
    A B C D E F G H I

    
    SERVICE DELIVERY LEVEL

28. Send clients to or receive homeless clients from this agency [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
29. Co-locate staff and/or services in the same location with this agency [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
30. Cooperate in serving clients [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
31. Communicate clearly with this agency in serving clients [Entry Field] A,[Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
32. Agree on goals with this agency for serving clients [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I

    
    LEADERSHIP LEVEL

33. Send funds to this agency [Entry Field] A, [Entry Field] B, [Entry Field] C,[Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H,[Entry Field] I
34. Influence flow of funds to agency [Entry Field] A, [Entry Field] B,[Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
35. Receive funds from this agency [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
36. To what extent does this agency influence the flow of funds to you [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E,[Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
37. Share data systems with this agency for assessing needs & delivering services [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D,[Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
38. Cooperate with this agency in planning [Entry Field] A, [Entry Field] B,[Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
39. Communicate clearly with this agency in planning [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I

    
    BOTH SERVICE DELIVERY & LEADERSHIP LEVELS

40. Send information to or receive information from this agency for coordination, control, planning, or evaluation [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
41. Have written documents (e.g., MOUs) or flow charts specifying working relationship with this agency [Entry Field] A, [Entry Field] B, [Entry Field] C, [Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I
42. Trust this agency [Entry Field] A, [Entry Field] B, [Entry Field] C,[Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H,[Entry Field] I
43. Respect this agency [Entry Field] A, [Entry Field] B, [Entry Field] C,[Entry Field] D, [Entry Field] E, [Entry Field] F, [Entry Field] G, [Entry Field] H, [Entry Field] I

    
    STRATEGIES TO IMPROVE SYSTEM INTEGRATION

44. To what extent are the following strategies to improve the integration of services for chronically homeless individuals currently in place either within the CHI network or through its involvement in a larger coalition? Note that in this question we are asking about structures that currently exist either within the CHI or within the larger community.
    • Organizing/coordinating body focused on ending chronic homelessness[Entry Field]0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
    • Strategic plan [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      ( Plan in place for producing permanent supported housing and other service configurations needed to end chronic homelessness.)
    • Data collection and feedback loop [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Data is not just collected with respect to individual clients but also on things like street counts to track success in ending chronic homelessness and to guide changes in program activities. )
    • Interagency coordinating body [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Group composed of representatives from multiple agencies brought together to address common concerns.)
    • Systems integration coordination position [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (A specific staff position focused on systems integration activities, identifying stakeholders, staffing inter- agency coalition meetings, acting as a liaison to other systems, coordinating the development of service contracts and joint proposals, and providing overall assistance for implementing systems integration.)
    • Interagency agreements / memorandums of understanding [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Formal and informal agreements among agencies that may include agreements to collaborate, make or accept referrals, share client information, or coordinate services.)
    • Interagency management information systems / client tracking systems [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Systems that promote interagency sharing of information, simplify interagency referrals, minimize paperwork, reduce duplication of services, and facilitate client access to services. )
    • Consolidation of programs / agencies [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      ( Combining multiple programs or agencies under central administrative structure in order to better integrate service delivery.
    • Pooled / joint funding [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      ( Combining or layering funds to create new resources or services to support CHI initiatives. )
    • Stable funding sources (new or existing) [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (State and/or local funding streams (e.g., housing trust fund, housing tax levy, tax increment financing) committed to producing more permanent supported housing, including capital, operating, and/or supportive services funding.
    • Funnel agency at the state or local level [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (An agency that receives multiple funding streams and disburses funding to housing and service providers through single consolidated applications.)
    • Expanding eligibility by changing rules or using special waivers [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Changing laws, rules and regulations, or eligibility criteria to make it easier for disabled homeless people to qualify for supports such as housing vouchers, SSI, or mental health services.)
    • Flexible funding [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Flexible non-categorical funding used to purchase expertise, fill gaps or coordinate the acquisition of additional resources to further systems-integration.)
    • Cross-training [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Training of staff about the objectives, procedures, and services available at other agencies.)
    • Co-location of services [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (A multi-service center in a single location (or mobile unit) providing a variety of services.)
    • Interagency service delivery team [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      ( An interdisciplinary service delivery team of staff from different agencies. These teams are designed to provide multiple services to address the needs of clients with multiple issues in an integrated manner.)
    • Dually qualified staff or agencies (mental health and substance abuse) [Entry Field]0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      ( Staff and/or agencies qualified & trained to address both MH and SA needs of people with both problems.)
    • Uniform applications, eligibility criteria, and intake assessments [Entry Field]0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (A standard process or form containing information used by participating agencies that an individual completes only once to apply for or receive services.)
    • Single entry point (at least for all singles and another for all families) [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (All people go to a single location to gain entry to emergency shelter, and possibly to transitional and permanent supportive housing.)
    • Real-time tracking of unit and service availability [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (On-line, real-time system that can be checked to see which programs have empty beds/housing units or service slots (e.g., for substance abuse treatment), and then arrange for clients to access.)

    
    PRACTICES

45. To what extent are the following services provided as part of the Chronic Homeless Initiative or available to its clients?
    • Outreach and engagement [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Meets immediate and basic needs for food, clothing, and shelter; non-threatening, flexible approach to engage and connect people to needed services).
    • Housing with appropriate supports [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      ( Includes a range of options from transitional to permanent and supportive housing; combines affordable, independent housing with flexible, supportive services).
    • Multidisciplinary treatment teams / intensive case management (ACT) [Entry Field]0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Provides or arranges for an individual’s clinical, housing, and other rehabilitation needs; features low caseloads (10-15:1) and 24-hour service availability).
    • Integrated treatment for co-occurring disorders [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Features coordinated clinical treatment of both psychiatric and substance use disorders within a team structure).
    • Motivational interventions / stages of change models [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Helps prepare individuals for active treatment; incorporates relapse prevention strategies; must be matched to an individual’s stage of recovery).
    • Modified therapeutic communities [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (View the community as the therapeutic method for recovery from substance abuse; have been successfully adapted for people who are homeless and people with co-occurring mental disorders).
    • Self-help programs (12-step only) [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Often include the 12-step method, with a focus on personal responsibility; may provide an important source of support for people who are homeless).
    • Involvement of consumers and recovering persons (other than 12-step) [Entry Field]0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      ( Can serve as positive role models, help reduce stigma, and make good team members; should be actively involved in the planning and delivery of services).
    • Prevention services [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      ( Reduce risk factors for chronic homelessness and enhance protective factors; include supportive services in housing, discharge planning, and additional support during transition periods).
    • No demand housing [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (No demand, unlimited stay housing, not requiring sobriety (i.e., either damp or wet)).
    • Housing first model of supportive housing [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Direct placement from chronic homelessness to permanent housing; unlimited length of stay; provision of flexible, individualized supportive services; maximum client choice; normalized settings (e.g., mixed income, resident population)).
    • Opiate substitution [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
      (Methadone maintenance)
    • Family psycho education [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
    • Individual placement and support (IPS) model of vocational rehabilitation [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
    • Psychopharmacology [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
    • Preventive primary care [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
46. Is there a specific effort to coordinate substance abuse and mental health services as part of the CHI network?
    1. Within an integrated team [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
    2. Between distinct substance abuse and mental health agencies [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK

    
    LARGER HOMELESS COALITION
    
    These last few questions are about the overarching coalition(s) with which the Chronic Homelessness Initiative agencies has been involved during the past year.

47. Is there a Department of Housing and Urban Development continuum of
    care currently operating in your community? [Entry Field] 0 = No, 1 = Yes, 9 = DK
48. Have the agencies participating in the Chronic Homelessness Initiative been
    part of one or more broader homelessness coalitions during the past year, including a HUD continuum of care)? [Entry Field] 0 = No, 1 = Yes, 9 = DK (If ‘No’, END interview.)
49. What is the name of the homelessness coalition most strongly associated with the Chronic Homelessness Initiative during the past year? [Entry Field]
50. How important has this coalition been in the development of the Chronic Homelessness Initiative application? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very, 9 = DK
51. How important will (is) this coalition be to the implementation of the Chronic Homelessness Initiative? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very, 9 = DK
52. To what extent does it provide financial or in-kind resources for the implementation of the Chronic Homelessness Initiative? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = A lot, 9 = DK
53. How influential will (is) the coalition be in shaping the goals of the Chronic Homelessness Initiative? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very, 9 = DK
54. How important will (are) the institutional and political support provided by the coalition be for theimplementation of the Chronic Homelessness Initiative? [Entry Field] 0 = Not at all, 1 = A little, 2 = Somewhat, 3 = Very, 9 = DK

This concludes the interview. Thank you for your time. We will contact you in about 12 months to schedule a time for our next interview. Do you have any questions you would like to ask me before we say goodbye? [Answer any questions.] Alright, then, I look forward to talking with you next year. Goodbye.

Footnotes

2. Model fidelity refers to the degree to which a program is implemented in conformance with an evidence-based treatment model (such as, Assertive Community Treatment), i.e. has elements and levels of intensity that experts have found to be part of effective models and does not have elements that are not thought to be part of effective models.

3. The PROC MIXED procedure of the SAS® software system Version 6.12 (SAS Institute, Cary, NC) was used for this analysis. Random effects were modeled for site in models in which the dependent measures were considered at the agency level and with respect to both site and agency for those models in which the dependent measure was at the dyad level.

4. The decline in the number of respondents by 10 from wave one to wave three at five sites, a 16.7% decline overall, primarily reflects the integration of mental health and substance abuse programs (60%) but also weakening involvement of general health programs (30%), and a consolidation of VA facilities at one site (10%).

5. An alpha level of less than 0.15 was used to determine statistical significance for these analyses because of the relatively small number of observations for testing the significant interactions between site/agency and time.

6. The existence of a fiscal relationship between two agencies was indicated by either the transfer of funds between the two agencies or the ability of one agency to influence the flow of funds to the other agency from third parties.