Alvin S. Mares, Ph.D., M.S.W.
Project Director, VA Northeast Program Evaluation Center
Robert A. Rosenheck, M.D.
Director, VA Northeast Program Evaluation Center
Northeast Program Evaluation Center (NEPEC)
VA Connecticut Healthcare System
February 26, 2007
This report is available on the Internet at:
Printer Friendly Version in PDF format (93 pages)
How to Obtain a Printed Copy
Executive Summary (in PDF format - 4 pages)
Peggy Halpern, Paul Dornan, Pete Dougherty, Anne Fletcher, Cynthia High, Mark Johnston, Robyn Raysor, and Gay Koerber participated in the Federal Funders CICH Evaluation Group representing HUD, DHHS, and VA provided essential oversight and review on earlier drafts of this paper.
We wish to acknowledge specifically the CICH evaluation site coordinators who coordinated data collection at their sites: Joyce Jones and Daniel White (Chattanooga), Eugene Herskovic (Chicago), JuanitaWilson (Columbus), Richard DiBlasio (Denver), Daniel Robbin and Elaine Stein (Ft. Lauderdale), John Nakashima (Los Angeles), Phyllis Larimore (Martinez), Julie Irwin (New York), Vincent Kane and Kimberly Lewis (Philadelphia), Lawrence Brennan (Portland), and Charlene Nason (San Francisco).
Brandi Williams coordinated data management at the VA Northeast Program
Evaluation Center (NEPEC) and Dennis Thompson provided computer programming support.
It has been estimated that 637,000 persons in the United States experience homelessness in any given week, and 2.1 million persons experience homelessness in a given year (Burt et al, 2001). While most of these persons experience homelessness for only a brief period of time, an estimated one-in-ten (10%) have experienced chronic homelessness, defined as those who experience homelessness characterized by either greater duration or more frequent of episodes (Kuhn & Culhane, 1998). Many of the estimated 150,000-200,000 persons experiencing chronic homelessness appear to have disabling health and or behavioral health problems (NOFA, 2003; SAMHSA, 2003). An estimated 40% of chronically homeless individuals have substance abuse problems, 25% have a disabling physical health problem, and 20% have a serious mental health problem (Culhane et al, 2001).
The major legislative impetus for federal agencies addressing the problem of homelessness, and more recently chronic homelessness, is the Stewart B. McKinney Homeless Assistance Act of 1987 (P.L. 100-77) known today as the McKinney-Vento Act. The Act, and amendments added in 1988, 1990, 1992, and 1994, include provisions designed to provide primary health care and mental health care for homeless Americans, and has encouraged diverse federal agencies to develop targeted service programs over the past 20 years. To encourage federal leadership in addressing the problem of homelessness, the Act established the U.S. Interagency Council on the Homeless (now the Interagency Council on Homelessness), which is currently comprised of the heads of 13 federal cabinet departments (Housing and Urban Development (HUD), Agriculture, Commerce, Defense, Education, Energy, Health and Human Services (HHS), Homeland Security, Interior, Justice, Labor, Transportation, and Veterans Affairs (VA)), as well as affiliated agencies (General Services Administration, Social Security Administration, United States Postal Service, Office of Management and Budget, USA Freedom Corps, and the White House Office of Faith-based Community Initiatives) (ICH, 2006).
In response to the goal of eliminating chronic homelessness, and the request that federal agencies increase their level of collaboration to accomplish this goal, members of the Interagency Council developed a new federal initiative entitled, Collaborative Initiative to Help End Chronic Homelessness (CICH). Through this program, persons experiencing chronic homelessness receive permanent supported housing funded by HUD, and supportive primary healthcare and mental health services provided by the Health Resources Services Administration (HRSA), the Substance Abuse and Mental Health Services Administrations (SAMHSA) of DHHS, and by the Veterans Health Administration (VHA) of VA (NOFA, 2003). A chronically homelessness person is defined, in this initiative, as "an unaccompanied homeless individual with a disabling condition who has either been continuously homeless for 1 year or more or has had at least four episodes of homelessness in the past 3 years".
The key components of the CICH intervention involve: 1) providing comprehensive primary health, mental health, and substance abuse treatment services linked to housing; 2) creating additional permanent housing; 3) increasing the use of mainstream resources that pay for services and treatment for this population; 4) replicating service, treatment and housing models that have proven to be effective ( e.g. Stein & Test, 1980; Tsemberis & Eisenberg, 2000); and 5) supporting the development of infrastructures that sustain the housing, services and treatments and interorganizational partnerhips beyond the designated CICH funding period.
Out of the more than 100 communities that applied for CICH funds, 11 were selected in October 2003 to receive funding. Funding varies by federal agency and types of service provided. HUD provided 5 years of funding to those sites with Shelter Plus Care programs and 3 years of funding to sites implementing Supportive Housing Programs. DHHS provided funding for 3 years for substance abuse, mental health and primary care services. The 11 communities included Chattanooga, TN; Chicago, IL; Columbus, OH; Denver, CO; Ft. Lauderdale, FL; Los Angeles, CA; Martinez, CA; New York, NY; Philadelphia, PA; Portland, OR; and, San Francisco, CA.
Each of these communities (sites) developed a comprehensive plan to end or reduce the prevalence of chronic homelessness in their community through the development of sustainable, cost-effective partnerships among providers in the private and public sector. The specifics of these plans vary across communities but each plan includes strategies for providing permanent housing, linking comprehensive supports with housing, increasing the use of mainstream services. integrating systems and services, and, ensuring the sustainability of these efforts beyond the funding period. A brief summary of programmatic efforts at each site is presented in Table 1 (National Technical Assistance Center [NTAC], 2006)
The three federal agencies sponsoring the Initiative (HUD, DHHS & VA) enlisted the VA Northeast Program Evaluation Center (NEPEC) to conduct a national evaluation of CICH client outcomes to monitor the implementation and effectiveness of the $55 million Initiative ($35 million funding in 2003, with $20 million added in subsequent years) by using a common evaluation methodology across all 11 sites.
The goal of the national evaluation is to provide a site-by-site description of program implementation, as well as descriptive information on clients served; services received; longitudinal housing quality, stability, and satisfaction; and, client outcomes in health and functional domains. Outcome data from the evaluation were provided to the sites throughout the implementation of the program to guide program development, and have been provided quarterly to allow ongoing monitoring of service delivery and outcomes. Monthly site-level statistics were provided to local CICH grant recipients on the implementation of evaluation procedures as well as to federal sponsors, beginning in May 2004, and updated longitudinal client outcome statistics were provided beginning in January 2005 every 3 months. These data will also form the basis for the evaluation of program accomplishments after it is completed.
This report presents data on both services and client outcomes from the first two years of CICH program operation. It also establishes the definitions of measures and general format of analyses for the final summary report, and addresses the following series of related questions:
[ Go to Contents ]
During the first two years of program operation (i.e., through April 2006), 1,430 homeless people were formally screened for enrollment in CICH, with substantial variation between sites ranging from 49 in Ft. Lauderdale to 476 in Martinez (Table 2). Of these, 1,242 (87%) were identified as having been enrolled into the CICH clinical program, with enrollments rates ranging from 38% in New York to 100% in Denver, Ft. Lauderdale and Philadelphia. All of those enrolled into CICH were to be invited to participate in the national evaluation. Among the 1,242 enrolled into the program nationally, 734 (59%) gave written informed consent to participate in the national evaluation, again with substantial variation across sites ranging from 11% in Martinez to 98% in Ft. Lauderdale. All those who provided written informed consent participated in data collection for the national evaluation.
Participation in the national evaluation was completely voluntary, and did not influence receipt of housing or services provided through the Initiative. Informed consent procedures were approved by Institutional Review Boards (IRBs) at the parent site and at each participating site.
During the second year of the program, from February 2005 through March 2006, a group of 118 comparison group subjects were recruited from five CICH sites: Chattanooga (N=19), Los Angeles (N=18), Martinez (N=32), New York (N=25), and Portland (N=24). Research staff at each of these sites were able to identify one or two portals of entry into existing housing, case management or supportive services programs serving chronically homeless persons in the same local area served by the CICH program.
The portals of entry for participating sites are identified below:
Local research staff, in cooperation with the staff of the housing and homeless service provider portals of entry identified above, recruited a convenience sample of chronically homeless adults in each of these communities that were intended to be socio-demographically and clinically similar to CICH clients (based on local research staff's experience recruiting CICH clients during the prior year), yet who were receiving some lesser combination of permanent housing, intensive case management, and access to healthcare services provided to CICH clients. Thus, comparison group subjects were intended to be chronically homeless adults receiving "usually available" housing resources and supportive services in each of the five communities.
The recruitment of a comparison group was approved by the same local IRBs which approved the recruitment of CICH clients to participate in the national evaluation. Written informed consent was provided by all comparison group subjects, who received the same remuneration of $15 per interview as CICH client subjects. Also, the same baseline and follow-up assessment forms and data collection procedures were used for both groups.
The remaining six CICH sites were unable to recruit comparison group samples due to local IRB, research staffing, and homeless service provider issues/challenges.
Client data were collected by full-time VA employees serving as "evaluation assistants," one at each site. These evaluation assistants were responsible for recruiting participants, collecting screening/intake forms completed by case managers, and administering baseline assessment interviews at entry into the formal program evaluation, along with quarterly follow-up assessment interviews for up to 3 years. Follow-up interviews were to be administered regardless of clients' housing or treatment status i.e., evaluation assistants were to continue the administration of follow-up interviews to clients who remained engaged in treatment, as well as those who discontinued participation in either housing or case management services throughout the Initiative.
Data collection began at the start of program initiation at each site, from March August 2004, and is intended to continue through September 2007. The data presented in this report were collected from the start of the project through mid-May, 2006, and represent 97% of all anticipated 3-month data, 93% of 6-month data, 88% of 9-month data, and 81% of all 12-month data to be collected. Thus we have limited 3-year data, but have nearly complete 1-year outcome data that have been collected and are available for presentation in this preliminary report.
Because comparison group recruitment began approximately one year after CICH client recruitment and lasted an average of 6 months (range 2 months in New York to 12 months in Martinez), we extended the data collection "cut-point" from mid-May 2006 to early November 2006 when comparing the two groups to provide enough 9 and 12-month follow-up observations for preliminary multivariate analyses of longitudinal patterns of service use and outcomes during the first year of CICH program operation.
A brief set of individual characteristics were measured through a structured screening form administered at first contact, prior to enrollment into the program. This form also recorded the date of enrollment in the program.
Further data were obtained during the baseline interview among those who agreed to participate in the evaluation. These data were collected to allow comparisons between people screened but not enrolled into the program and those who were enrolled, both program-wide and at each participating site, as well as a comparison of enrolled clients who did and who did not give consent to participate in the evaluation.
Due to the intensive and comprehensive nature of the Collaborative Initiative and the heterogeneity of the target population, an extensive array of service use and client outcome measures were chosen to document the diverse processes and outcomes of the program.
Receipt of four types of services typically needed by chronically homeless people were documented: primary health care services, mental health & substance abuse treatment, case management, and the overall integration of service delivery. The following seven client outcome domains were examined: housing status, community adjustment, mental health status, substance use, physical health status, income, and health care costs.
A brief explanation of each of these measures follows.
CICH intake and outreach staff completed a structured form on each person screened, which documented: 1) basic socio-demographic information (i.e., age, gender, and race/ethnicity); 2) eligibility characteristics (i.e., single individual vs. parent, and type of chronic homelessness experience a current episode of homelessness lasting a year or longer vs. 4 or more episodes of homelessness during the past 3 years); 3) each of three disabling condition(s) (i.e., mental health, substance abuse or medical) identified at screening and confirmed by evaluation staff by clarifying those conditions noted by clinicians at screening with program intake staff and by asking clients during the baseline assessment interview; 4) outreach location and the agency initiating outreach; 5) outreach clinician observations of clinical problems; 6) response to early interactions with intake/outreach staff; and, 7) date enrolled into the program (if applicable). Measures of time (days) from screening to enrollment (among those enrolled into the program), and time from enrollment to the baseline assessment interview (among CICH clients) were also documented.
Supplemental socio-demographic and clinical information (e.g., veteran status, disabling condition(s) were collected following written informed consent, along with service use and client outcome data.
First, CICH clients were asked whether they had a "usual health care provider", and whether they had health insurance?
Then the number of routine, preventive healthcare procedures received during the past year was assessed from a list of 12 gender-neutral procedures, plus either four male-specific procedures or two female-specific procedures (Gelberg, 2003). Examples of gender-neutral procedures include measurement of height, weight, blood pressure, cholesterol, hearing and vision, testing for diabetes, tuberculosis, hepatitis, and routine dental care. Male-specific procedures included prostate exam, rectal exam, colonoscopy, and PSA testing. Female-specific procedures included PAP testing and breast exam.
Clients reporting unhealthy behaviors were also asked the number of such behaviors that they had discussed with a healthcare professional during the previous year. These behaviors included drinking alcohol among drinkers, smoking among smokers, and diet/nutrition among those who were obese at baseline.
The total number of outpatient medical visits made during the past 3 months was also included as a primary health treatment measure.
Finally, the trust in physician scale (Anderson & Dedrick, 1990), an 11-item measure, was used to assess the level of trust felt by a patient with his/her primary doctor/physical healthcare provider. Item responses range from 1=totally disagree to 5=totally agree. The scale score is computed as the mean response to these 11 items, with a higher score reflecting greater trust felt by CICH clients and their primary provider (alpha=.91). Sample items include: "My health care provider is usually considerate of my needs and puts them first", "I trust my health care provider so much that I always try to follow his/her advice," and "If my health care provider tells me something is so, then it must be true".
Clients were further asked whether they could identify a primary mental health or substance abuse treatment provider, as well as the total number of outpatient mental health visits and the total number of outpatient substance abuse treatment visits received during the previous 3 months. A fourth measure addressed participation in self-help groups (i.e., Alcoholics Anonymous or Narcotics Anonymous)
A 7-item therapeutic alliance scale was used to measure the strength of the therapeutic relationship experienced by CICH clients with their primary mental health or substance abuse provider (Neale & Rosenheck, 1995). Scale scores range from 0 to 5 reflecting greater strength of relationship, and were calculated as the average response to the 7 items (alpha=.94; mean=4.4; range 0-6). For example, clients were asked, "How often does your provider perceive accurately what your goals are?", and "How often are the goals of your work with your provider important to you?".
Finally, clients' experience of personal choice in selecting mental health or substance abuse services was measured using a 5-item "consumer choice" scale (Monahan et. al., 2005), which was computed as the average response (alpha=.89; mean=4.0; range=1-5), ranging from 1=strongly disagree to 5=strongly agree, with higher scores reflecting greater client choice. The scale included the following items: 1) I felt free to do what I wanted about going for treatment; 2) I chose to go for treatment; 3) It was my idea to obtain treatment; 4) I had a lot of control over whether I went for treatment; and, 5) I had more influence than any one else on whether I went for treatment.
Clients were also asked whether they could identify a primary case manager, and whether they were visited by a case manager in a community setting (i.e., either at home or at some other place in the community other than a service agency or healthcare facility setting) during the previous 3 months.
Clients were further asked whether they had a money manager ("a person or organization which helps you manage your money") and whether they had had any contact with their landlord, either in-person or by telephone, during the past 3 months? Money management has been identified as an important ingredient in the approach to helping homeless people developed by Tsemberis and colleagues (2000) and landlord-tenant relationships have been found to be associated with housing outcomes among persons with mental health and/or substance abuse problems (Kloos et. al., 2002).
Both objective and subjective measures were used to evaluate the integration or coordination of diverse CICH services. Services integration is defined here to be the extent to which the key components characterizing the CICH intervention (listed below) were provided to clients. An objective measure of overall services integration was based on calculation of the proportion of total component services received by clients using a series of six dichotomous service component measures including: 1) independent housing 2) case management, 3) general medical care, 4) substance abuse treatment, 5) mental health services, and 6) VA services. If the client did not have need of a particular service or was not eligible to receive VA services, the score for that particular item was coded as missing. Scores on non-missing values were averaged and the total ranged from 0 to 100%, with a higher score representing more fully integrated service delivery (overall mean at baseline=.64; sd=.23).
A second objective measure of overall service delivery represented the total number of outpatient health visits of all kinds received during the previous 3 months. This measure was calculated as the sum of medical visits, mental health visits, and substance abuse treatment visits. Clients were further asked to estimate the total number of different individual service providers assisting them during the past 3 months.
To supplement these objective measures, a 5-item subjective scale was developed to measure the extent to which the delivery of these services was perceived to be well coordinated or fragmented. The five component items included one which asked about the coordination of services ("How often do providers work together to coordinate your care?", and four which addressed fragmentation (i.e., "How often is one provider unaware of information about your care that another provider has?", and "How often do these providers seem unaware of what the others are doing for you?", and "How often do you have to tell the same information to several providers?"). The response set to these items was 0=rarely, 1=sometimes, and 2=often, and the scale score is the average response across the 5 items, after recoding fragmentation items so that higher scores reflect a higher degree of coordination of services and less fragmentation (alpha=.80; mean=1.3; range=0-2 over all time points).
The primary goal of the Collaborative Initiative was the rapid and sustained placement of chronically homeless people into long-term housing. Clients were asked at each interview the number of days during the past 3 months that they were housed in each of nine settings, as well as where they were residing at the time of each assessment. The number of days housed was defined as living in their own place, someone else's place, or in an SRO hotel or boarding home. SRO hotels were considered residences because some sites used such housing as the primary housing resource for CICH clients (e.g., Los Angeles and San Francisco). Nights spent in shelters, outdoors, or in vehicles or abandoned buildings were classified as representing "homeless" housing status.
Clients who were living in their own places were also asked to report their level of satisfaction with their housing using a 20-item housing satisfaction scale developed by Tsemberis and colleagues (2003), as part of the SAMHSA Supported Housing Initiative (CMHS, 2001). Responses to these items ranged from 1=very dissatisfied to 5=very satisfied, so the higher the score, the greater the satisfaction with housing. The scale score was calculated as the mean of the responses to these 20 items (alpha=.89). Sample items include, "The amount of choice you had over the place you live", "How close you live to family and friends", "The safety of your neighborhood", and "The amount of privacy you have".
To evaluate how well CICH clients were integrated into and engaged in community life, they were asked whether they had participated in each of 16 common activities (e.g., visiting with close friends/relatives/neighbors, going to a grocery store, restaurant, retail store, bank, movie, library, park, or reading a newspaper) during the previous 2 weeks (Katz, 1963). A "community involvement" scale was then calculated as the total number of these activities, and ranged from 0 to 16, with a higher score indicating greater participation in community activities.
Social support networks were assessed by questions asking the number of types of persons who would be available to help them about three different types of assistance: a short-term loan of $100, a ride to an appointment, or someone to talk with if they felt suicidal (Vaux et. al., 1987).
Additional single-item measures addressed a) whether clients knew any of their neighbors well, b) the number of days spent in jail during the past 3 months, and c) satisfaction with life overall (subjective quality of life), scored on a 7-point terrible=1 to delighted=7 scale, with a higher score indicating greater satisfaction with life. (Lehman, 1988).
CICH client income was expected to rise as a result of participating in the program, both through increased access to public support benefit payments and through employment. Clients were asked whether they had received any of several types of public support income during the past month, and if so, the amount of such income. Information on days of employment and employment income were also obtained, along with informal types of income. Responses to these items were summed to create a measure of total income.
The Medical Outcomes Study Short Form (SF)-12 mental health subscale (Ware et. al., 1998), three subscales from the Brief Symptom Inventory (BSI) (Derogatis & Spencer, 1982), and an observed psychotic behavior rating scale (Dohrenwend, 1982) were used to evaluate mental health status. The SF-12 mental and physical health subscales assess the overall level of functioning in their respective domains, with scores ranging from 0 to 100, and a score of 50 representing normal level of functioning in the general population. Each 10 point-interval represents one standard deviation (above or below) the general population norm.
Three BSI subscales were used reflecting major domains of subjective distress: psychoticism (e.g. hallucinations, delusional beliefs, disorganized thinking), depression, and anxiety. The BSI score presented in this report is the mean value for these three sub-scales, ranging from 0=never experience symptom to 4=very often experience symptom.
Finally, a measure of observed psychotic behaviors included 10 types of psychotic behavior observable by evaluation staff during the course of baseline and follow-up interviews (e.g., auditory or visual hallucinations, delusions, agitation/aggression, inappropriate behavior or speech, incoherent speech). Each of these behaviors was coded 0=not at all to 3=a lot, based on staff observations, and the total scale score was computed as the average score across these 10 items (alpha=.76; mean=0.2; range=0-2.6 over all time points).
Items from the Addiction Severity Index (ASI) documented the number of days each client drank to intoxication and whether they had used any illicit drugs during the previous month. Alcohol and drug sub-scales (McLellan et. al., 1980) measured alcohol and drug use problems using multiple items combined in a standard comparable score ranging from 0 to 1, and which included items such as, "Do you feel like you have a problem with alcohol now?", "How many days in the past 30 have you experienced drug problems?", and "How troubled or bothered have you been in the past 30 days by your own alcohol/drug problems?". A higher score on an these ASI sub-scales reflects a greater, more serious substance use problem.
Service costs were estimated for four aggregated types of care: medical/dental treatment, mental health services, substance abuse services, and the total for all three types of services. These estimates were computed by multiplying the number of visits/days of care reported by standard estimates of the unit cost of each type of care. Unit costs were estimated on the basis of data average unit cost data compiled for a recent NIMH funded cost effectiveness study of treatment of schizophrenia (Rosenheck et. al., 2006) as follows:
For example, if during the previous 3 months a CICH client reported spending three days in the hospital, two days in the emergency room, and had six outpatient visits for a physical health problem, and one dental visit, then his/her medical/dental quarterly service cost for that quarter would have been $6,260 ((3*$1,866) + (2*$96) + (6*$70) + (1*$50)).
Total health costs were also sub-grouped across types of service into inpatient and outpatient costs. Inpatient costs were calculated as the sum of the following five service costs: inpatient medical, mental health, substance abuse, day hospital, and residential treatment. Outpatient costs were similarly calculated as the sum of each of the various types of outpatient costs multiplied by their unit costs listed above.
Basic demographic and clinical characteristics of all individuals screened for possible enrollment into the program are presented, for both the entire sample and for each site. Analysis of variance and chi square tests are used to identify the significance of differences in individual characteristics across sites at screening.
Next, demographic and clinical characteristics of individuals screened but not enrolled into the project are compared with those of individuals who enrolled in the project and with those who gave consent to participate in the evaluation, again using data from the screening form with the significance of differences evaluated by chi-square and independent samples t-tests.
We next present baseline characteristics of all CICH clients participating in the evaluation with a comparison of differences across sites, using data obtained from the baseline interview. The significance of differences across sites were evaluated using one-way analysis of variance (ANOVA) and chi-square tests.
Repeated measures mixed regression models then were used to identify changes in service use and client outcomes. Mixed model linear regression is a statistical technique used to analyze longitudinal outcome data in which follow-up observation data points are not independent (i.e. they are correlated because they pertain to the same individual/client). Thus, a distinctive feature of these models is that they adjust the standard errors, the spread of follow-up data around the group mean for the correlation of data within individual clients (Bryk & Raudenbush, 1992).
The mixed linear regression models used in this report generally included one or more class variables, representing time (e.g., baseline, 3-month, 6-month, 9-month follow-up) and client sub-groups of interest (e.g., housed at baseline vs. not housed, male vs. female, etc.). They also included one or more covariates representing potentially confounding baseline characteristics (e.g., site or client socio-demographic characteristics bivariately associated with service use and outcomes measures, including the baseline values of client outcome measures). We present least square means which are adjusted for these covariates. Some mixed regression models examined the main effect of time alone, while others examined the 2-way interaction effect of time and various client sub-groups. Main effect statistics representing time are used to determine whether there was significant change in a given measure over the period of the evaluation, while interaction statistics compare patterns of change over time between client sub-groups (e.g., whether clients housed at baseline or other subgroups showed a greater or lesser rate of improvement over time than other participants).
Two sets of mixed models were developed. The first set of models was developed to examine changes in the use of services and in client outcomes over the 12-month follow-up period for all evaluation subjects, without considering whether they were housed at baseline or not. For each measure, a single class variable represented the main effect of time and was coded 0 for the baseline observation, and 1 through 4 for 3-month, 6-month, 9-month, and 12-month observations, respectively. Covariates used included 10 site dummy codes to adjust for site effects for all service use and outcome measures. Covariates used with client outcome measures also included the baseline value of the measure, and 11 eligibility and baseline socio-demographic characteristics. Least square means, adjusted for covariates and site effects are reported for each time point. Statistics indicating whether significant trends (upward or downward) were observed among measures of service use and client outcomes over time are also reported.
Mixed models with weighted observations (marginal structural models) were used to examine the possible influence of attrition (loss to follow-up) on service use and outcomes (Robins & Finkelstein, 2000). Logistic regression was first used to model successful follow-up at each time point, using the set of 11 eligibility and baseline socio-demographic characteristics to estimate the predicted probability that each interview would be completed. Observations were then weighted by the inverse of the predicted probability that they would be completed so that interviews completed with people whose baseline characteristics suggest a low probability of follow up were given more weight than those more likely to be completed.
A second set of mixed regression models is presented to determine the extent to which housing status at baseline or other baseline characteristics were significantly associated with changes in service use and/or outcomes over time. In these analyses housing status at baseline was added as a second class variable, coded "1" for clients housed at baseline and "0" for others. Baseline housing status, group, and group x time interactions were both examined. Baseline differences between the housed and not housed groups identified as significant on bivariate analyses (using ANOVA analyses) were included as covariates. Least square means for each time point, for each housing status group, are reported, along with fixed effect statistics representing the significance of the interaction of housing group and time.
Differences in patterns of service use and outcomes across sites were evaluated using mixed models that included main effect for site, for time, and a term presenting site by time interaction.
To determine whether the differences in patterns of service use account for the differences in outcome across sites, we re-tested the significance of differences in outcomes across sites including measures of service use as covariates.
Bivariate differences on over 250 baseline characteristics between CICH client subjects (N=296) and comparison group (N=118) subjects at the five comparison group CICH sites were evaluated using independent samples t-tests. Significant differences between the two groups were found on 80 of the 253 baseline characteristics examined. Descriptive statistics were then run on these 80 baseline measures among the combined sample of CICH clients and comparison subjects (N=414) to identify measures having larger amounts of missing data. Eleven measures were found to have fewer than 380 observations, due largely to skip patterns within the baseline assessment form, and were therefore excluded from subsequent logistic regression analyses. Sixteen other baseline measures were also excluded due to redundancy with other variables. Logistic regression was then used to select a smaller, more parsimonious set of baseline covariates among the remaining 53 bivariately significant measures to be included in subsequent multivariate mixed model analyses. The 53 baseline measures were entered as a single block of independent variables, with group (coded as 1=CICH client, 2=comparison group subject) as the dependent variable. The stepwise method of selection was used, both forward and backward.
The stepwise forward selection method identified 10 measures in the final model. When stepwise backward selection method was used, the same 10 measures were identified along with four additional measures, for a total of 14 baseline covariates. These covariates included category of chronic homelessness (i.e., experiencing 4 or more episodes of homelessness during the previous 3 years); race/ethnicity (i.e., Asian/Pacific Islander); work history (i.e., primarily unemployed during the previous 3 years); access to healthcare (i.e., having health insurance and a usual source for medical care, and the total number of healthcare providers); mental health diagnoses (i.e., bipolar or anxiety disorder, and total number of mental health and substance abuse diagnoses); receiving supportive services (i.e., housing, vocational rehabilitation, case management visits in the community); number of days hospitalized; and, spirituality. On most measures comparison clients have less severe health problems and better community adjustment.
Multivariate mixed models were then used to evaluate differences between CICH and comparison clients in service use and client outcomes during the first 12-months of CICH program, as described earlier. The class variables in these mixed models included group, time, and the group*time interaction. Covariates included the baseline value of the dependent variables, four site dummy codes (with New York being the reference site), and the 14 baseline covariates identified above.
[ Go to Contents ]
Collaborative Initiative staff at the 11 sites screened a total of 1,430 persons for possible enrollment into the program, with substantial variation across sites (e.g. 49 at Ft. Lauderdale, 72 at Philadelphia, 146 screened at Los Angeles and 476 at Martinez (Table 2). On average 130 persons (sd=118) were screened per site. While each site was asked to complete a client screening form for each person evaluated for enrollment into the program, the implementation of screening procedures varied across sites. Differences in screening procedures most likely reflected differences in local goals and circumstances and most likely account for the variation observed in both the number of persons screened, the proportion of people screened who were enrolled into the program, and the numbers and proportions of those enrolled who participated in the evaluation.
Comparisons of individual characteristics of people screened across sites are notable. For example, at Chattanooga, Ft. Lauderdale, Martinez, and Portland a higher proportion of Caucasian (non-minority) persons were screened as compared with Chicago, New York and Philadelphia (Table 2). Whether this result is due to a higher proportion of Caucasians living in these areas or some other reason is not known.
Perhaps the most salient difference in target populations observed across sites was in the area of major diagnostic groups. New York, and to a lesser extent Chicago and Martinez, primarily screened individuals with substance abuse problems, in contrast to Ft. Lauderdale, Chattanooga & Columbus, which primarily screened those with mental health problems (Table 2).
Statistically significant bi-variate differences across sites were found on 44 of 48 screening measures. For this reason, as noted above, ten site dummy codes (coded as 0 or 1) with one site excluded as the reference condition, were included in subsequent models.
Overall, among those screened, the average age was 45 years, 74% were men, 45% were Caucasian, they all were individuals caring for themselves alone (i.e., no families were screened), and 69% had been homeless for over a year, with an additional 31% having experienced 4 or more episodes of homelessness within the past 3 years. Most of those screened had a substance abuse problem (77%) or a mental health problem (60%). About one-third of those screened had one or more disabling medical condition(s). Half of screening/outreach contacts occurred in a shelter or mission (26%), or on the streets or some other outdoor location (25%). Outreach activities were more often initiated by homeless service providers participating in the local CICH Collaborative (51%), than by the staff of the lead agency (27%) or through referrals by other agencies (10%). When approached by Collaborative outreach staff, nearly two-thirds of those screened (63%) expressed interest in receiving permanent housing and supportive services provided through the program.
Of those screened, all but 13% were subsequently documented to have been enrolled into the program (Table 2, third row from the bottom). Half of those screened (51%) both enrolled into the program and agreed to participate in the national evaluation (Table 2, bottom row), again with substantial variation across sites.
Many differences in screening characteristics were found between a) those screened who did not enroll in the program (N=188; 13%), b) those who enrolled into the program but who did not participate in the national evaluation (N=508; 36%), and c) those who agreed to participate in the evaluation (N=734; 51%) (Table 3). When compared with those screened but who did not enroll in the CICH program, those who enrolled were younger, more likely to be female; less likely to be black; and substantially more likely to have medical or mental health problems (Table 3). Since those enrolled into the program were more likely to have medical or mental health problems than those screened but not enrolled., "creaming", or selecting "healthier" people for enrollment into the program does not appear to have occurred.
Compared to those who enrolled but did not participate in the evaluation, those who enrolled into CICH and did participate in the evaluation were generally older, more likely to be male and black, far more likely to have medical or mental health problems (including higher rates of psychotic disorder or other serious mental illness) and to have alcohol abuse problems (Table 3). Moreover, participants in the evaluation were less likely to have been screened outdoors or at drop-in centers, and were more likely to be screened at soup kitchens or treatment programs.
Three months prior to the baseline assessment, the proportion of CICH clients who had spent at least one night in the nine housing arrangements examined were as follows: own place 29%; someone else's place 21%; hotel/SRO/boarding home 15%; halfway house 13%; transitional housing 12%; hospital 15%; jail 5%; shelter 53%; and, outdoors 43%. These percentages add up to greater than 100% because CICH clients lived in an average of 2.1 (sd=1.4) different housing arrangements during the 3-month period prior to the baseline assessment interview.
Baseline characteristics including program eligibility, demographic characteristics, use of outpatient services, and a variety of baseline health status measures are presented for both the entire sample of CICH clients (N=736), and for participants at each site (ranging from N=48 in Ft. Lauderdale to N=97 in Denver) (Table 4).
Evaluation participants were enrolled into the program within 8 days of being screened, on average (sd=49), and had their baseline assessments within 30 days (sd=49) of being enrolled into the program (Table 4, bottom two rows). Philadelphia and San Francisco were distinctive with respect to lag times both between screening and enrollment, and between enrollment and baseline assessment. Screening assessment dates for evaluation participants in Philadelphia were generally 40 days after their enrollment dates. In contrast, in San Francisco participants were generally screened 59 days before their enrollment dates. However, baseline assessments were administered to participants in San Francisco within just 12 days of program enrollment, compared with the 63 day average enrollment-baseline lag time in Philadelphia (Table 4). As with the variation in number and proportion of people screened and enrolled described above, unique recruitment and enrollment procedures developed in response to local conditions at each site and summarized previously (Table 1) may explain the variation observed across sites in enrollment and baseline assessment lag times.
A potentially important difference between sites is the percentage of subjects who were already housed and living in their own place at baseline (28% overall, ranging from 4% in Ft. Lauderdale, Philadelphia, and Portland, to 38% (Los Angeles), 41% (Denver), and 84% (San Francisco)). Since more than one in four participants were already in permanent independent housing before informed consent could be obtained (i.e. at the time the baseline assessment was administered) , we examined differences in baseline characteristics between subjects who were housed at baseline, and those not yet housed, as well in subsequent service use and outcomes.
Adjusting for site, those living in their own place at the time of the baseline interview were less likely to be veterans (18% vs. 35%), had a higher levels of mental functioning on the SF-12 (40.4 vs. 38.3), and were less distressed due to psychiatric symptoms (1.36 vs. 1.59) (Table 5), perhaps reflecting fewer impediments to entering into housing and/ or the positive short-term effects of being housed. These individual characteristics were also included as covariates in subsequent multivariate analyses to adjust for potentially confounding effect of housing status at baseline on longitudinal service use patterns and client outcomes.
Follow-up rates among all CICH clients participating in the evaluation during the first year of follow-up were as follows: 91% at 3-months, 88% at 6-months, 87% at 9-months, and 85% at 12-months. Nearly 2,400 follow-up assessments were completed during the first year of client follow-up (all of which took place between March 2004 and May 2006). Some interviews were only partially completed or have yet to be submitted (Table 6). Across the sites, 12-month follow-up rates ranged from 73% in Columbus and 77% in Chattanooga, to 93% in Martinez and 94% in Denver (Table 6).
Adjusting for site differences, statistically significant time effects were found on 15 of 19 service use measures, and 20 of 30 client outcome measures (Table 7) briefly summarized below:
Access to healthcare services and case management increased over the first 6 months as the proportion of participants who reported having a usual health care provider increased from 36% at baseline to 52% and 49% at 9 and 12-month follow-ups (Table 7). Other indicators of increased access to healthcare services included an increase in the average number of preventive procedures (from 7.4 at baseline to 8.4 at 9 and 12-months), and an increase in the average number of health behaviors discussed with a healthcare provider (from 3.3 at baseline to 3.9 at 12-month follow-up). The proportion of uninsured decreased from 21% at baseline to 12% at 6, 9 and 12-months thereafter.
Increasing proportions of clients also reported being visited by case managers in the community (45% at baseline to 71% and 67% at 9 and 12-months' follow-up), being provided money management assistance (19% at baseline to 31% at 12-months), and having contact with their landlords (70% at baseline to 75% at 12-months) (Table 7).
While no statistically significant change was found in the average number of outpatient medical services used over time, the average number of outpatient mental health visits increased from 3.5 at baseline to 4.1 at 3-months, and then declined to 2.9 at 12-months. The average number of substance use treatment visits declined from a peak of 5.0 at baseline to 2.9 at 9-months, and then increased modestly to 3.5 at 12-months.
The total number of outpatient visits for all services increased from 11.2 at baseline to 12.1 at 3-months, and then declined to 8.6 and 8.8 at 9 and 12-months, while. the proportion of the six key service components received by each client rose from 64% at baseline to 81% 3 months after entering the program and remained fairly constant thereafter (bottom line of upper panel of Table 7). This suggests that clients remained engaged in most aspects of the CICH intervention during their first year in the program, although use of outpatient services decreased after the first 3 month assessment.
The average number of individual providers decreased over time from 4.5 at 3-months to 4.0 at 9 and 12-months, while the index of subjectively experienced coordination of services (range 0-2) increased from 1.1 at baseline to 1.3 at 3-months and thereafter (Table 7).
CICH clients were housed quickly, and remained housed in one or more different settings thereafter. An examination of living arrangements among CICH clients at baseline and at each subsequent completed follow-up interview shows that 89% of CICH clients appeared to have been continuously housed throughout the 12-month follow-up period, where continuously housed is defined as living in their own place, in someone else's place, or in an SRO hotel at the time of the interview. In contrast, only 2% lacked housing at baseline and at all completed follow-up assessments thereafter, and 8% were housed during some quarterly assessments, and not housed during other assessments (data not shown). The average number of days housed in the previous 90 days increased dramatically from 18 at baseline to 68 at the 3-month follow-up, and rose steadily thereafter to 83 at the 12 month follow-up. The fact that housing satisfaction remained largely unchanged among those housed, averaging 4.1 (out of a total possible 5) at baseline and 4.0 thereafter, suggests that CICH clients were generally quite satisfied with their housing placements, and remained so (Table 7). Once housed, clients quickly got to know at least one of their neighbors well (increasing from 38% at baseline to 67% at 3-months, and then increasing to 76-77% thereafter). Subjective quality of life (i.e. life satisfaction) also increased rapidly, albeit modestly, from 4.3 at baseline to 4.6 at 3-months and thereafter (an increase of 7%). The community integration index (reflecting participation in everyday activities such as shopping or going to a restaurant) also increased from 6.8 at baseline to 7.2-7.3 at 6-months and thereafter (also a 7% increase).
One possible contributor to increased subjective quality of life and community integration is increased income. Average total income increased continuously from $380 at baseline to $579 at 12-months. The proportion of clients receiving any type of public support rose from 70% at baseline to 81%-85% at 3-months and thereafter. Consequently, average public support income rose by 55%, from $316 at baseline to $478 a year later (Table 7).
Mental health functioning also increased over time, from 38.8 at baseline to 39.9-40.6 at 3-months and thereafter (a 4% increase), while psychological distress decreased steadily and significantly over time from 1.53 at baseline to 1.29 at 12-months (a 16% decline).
Finally, total average quarterly treatment costs per person decreased by 51% over the first year of treatment, from $6,832 at baseline to $3,376 12-months later, reflecting substantial decreased use of inpatient services (Table 7, bottom panel). Medical and dental costs costs per quarter decreased from $3,219 at baseline to $1,512 at 12-months. Similarly, mental health quarterly costs decreased from $2,303 at baseline to $1,138 at 12-months, while substance use treatment costs per quarter decreased from $1,310 at baseline to $734 at 12-months. The primary source of the decline in these costs was reduced inpatient treatment which decreased by over $3,000 (from $5,776 at baseline to $2,677 a year later). The decrease in quarterly outpatient costs was significant, but smaller at about $350 from $1,056 to $698 during the first year of treatment (Table 7).
No significant changes were found on other outcome measures over time, including social support, days in jail, and level of physical health functioning.
Although data loss was modest during the first year, especially for a chronically homeless sample, we examined the extent to which changes in service use and outcomes described above may have been biased by the attrition rates presented in Table 6. For this analysis we used the "marginal statistical model method of Robbins et al., (2000, 1993) to create sampling weights equal to the inverse of the predicted probability for each client of completing a follow-up interview for each period of time. This probability was calculated using a logistic regression model in which seven basic eligibility characteristics (two homelessness categories and five disabling conditions, including dual mental health and substance abuse problem) and seven baseline socio-demographic characteristics (age, gender, education, race/ethnicity, marital status, veteran status, and duration of homelessness in lifetime) were entered as a single block of predictors.
We thus weighted observations from persons with similar socio-demographic and clinical profiles who were more likely to have missed a given follow-up interview more heavily. Mixed regression models using these weights were used to calculate the least square means presented in the non-weighted Table 7. Statistical significance test results changed for only 1 measure, illicit drug use. Although no difference in illicit drug use rates was found between non-weighted and weighted analyses, the later showed a small, but statistically significant decrease over time, whereas no significant change was found using unweighted analysis. (Table 8). Thus, attrition bias appears to be minimal, and subsequent analyses were run using the simpler and more straightforward non-weighted observations.
Because of concern that results could be biased by the fact that 28% of clients were already living in independent housing at the time of the baseline assessment (Table 4), we examined differences in service use and outcomes by this criterion. Adjusting for the baseline value of each measure, for site, and for significant bivariate differences between those housed and not housed at baseline (i.e., veteran status, mental health functioning, and psychiatric symptom distress) (Table 5), the only significant difference in service use was in the proportion of core CICH services received (i.e. overall service integration Table 9, last row) (71-74% vs. 80-83% after 3 months of entering the program). It thus appears that clients housed before informed consent and baseline assessment were obtained were somewhat less engaged in full array of CICH services.
Significant differences between clients housed at baseline and others were observed on 7 of 30 outcome measures, primarily the 4 housing outcome measures (Table 10).
Clients housed at baseline also had more days of drinking to intoxication and more serious alcohol problems than those not housed at baseline, but also showed higher levels of physical functioning (Table 10). Alcohol problems increased more among clients housed at baseline than others, possibly suggesting that rapid placement into permanent housing without sobriety requirements may have facilitated continued abuse of alcohol during the first year in the program. It is also possible, however, that rapid placement allowed clients with more serious addiction problems to be served. Nevertheless, outcomes on these measures were no longer statistically different from 6-12 months.
Housing status at baseline thus appears not to be a seriously confounding influence on changes in service use patterns and outcomes over time. We nevertheless included the "housing status at baseline" indicator as a covariate in subsequent multivariate analyses.
Mean service use levels and outcomes (Table 11) during the 12-month follow-up period were compared among the following five client sub-groups defined by critical baseline characteristics: 1) type of psychiatric disability (mental health vs. substance abuse vs. both), 2) age, 3) race/ethnicity, 4) incarceration history, 5) military service, and 6) gender.
Clients having both mental health and substance abuse problems (N=367; 52%) (hereafter referred to as "dual problem" clients) generally consumed higher levels of services than clients with substance abuse problems alone (N=137; 20%) (Table 11). Service use patterns of clients with mental health problems (N=170; 24%) and dual problem clients were similar, except that dual problem clients as one would expect, had more outpatient substance abuse visits quarterly (5.3 vs. 1.6 visits), and greater participation in Alcoholics Anonymous and Narcotics Anonymous (45% vs. 10%), than clients with mental health problems alone (Table 11).
Younger clients, i.e. under the age of 50 (N=462; 66%), were greater consumers of mental health services, compared with older clients who made greater use of medical services. Younger clients were also more likely to have a primary mental health/substance abuse treater (67% vs. 62%), had more outpatient mental health visits (3.8 vs. 3.0), and were more likely to have a primary case manager (28% vs. 24%) than older clients. Older clients, however, were more likely to have a usual physical health care provider (49% vs. 44%), had a greater number of preventive medical procedures (8.6 vs. 7.8), and had a greater number of outpatient medical visits (3.3 vs. 2.5) than younger clients (Table 11).
The only statistically significant racial/ethnic difference in service use was that minorities (N=439; 63%) consumed more outpatient medical visits during the 12-month follow-up period than Caucasian, non-Hispanic clients (3.0 vs. 2.3) (Table 11).
Clients who had been incarcerated in their lifetimes (N=476; 71%) prior to entering CICH used more outpatient health (11.2 vs. 8.0) and substance abuse visits (5.0 vs. 1.9) and participated more in AA/NA (37% vs. 22%) than clients without incarceration histories, who , in turn, were more likely to have a usual health care provider (52% vs. 43%) (Table 11).
Veterans (N=212; 30%) accessed higher levels of healthcare services than non-veterans. Veterans were less likely to report being uninsured (3% vs. 19%), more likely to have a usual health care provider (55% vs. 42%), received more outpatient visits overall (11.8 vs. 9.5), and had more substance abuse treatment visits (5.5 vs. 3.3), perhaps reflecting their access to VA services, an important component of the CICH initiative (Table 11).
Female clients (N=171; 24%) were more likely than men to report having a usual health care provider (52% vs. 44%), outpatient mental health visits (4.1 vs 3.3), and a greater number of preventive procedures (8.7 vs. 7.9). Additionally, they were more likely to have a primary mental health/substance abuse treater (69% vs. 64%), a primary case manager (30% vs. 26%), and a money manager (32% vs. 24%) than were male clients (Table 11).
Significant differences were found among psychiatric disability groups on 27 of 30 outcome measures examined. Compared with clients with substance abuse problems, clients having mental health problems were less satisfied with their housing and with life in general, participated less in community activities, were less likely to know their neighbors well, less likely to work, and more dependent on public assistance. As a result they had higher total incomes, were more disturbed by psychiatric symptoms, and demonstrated more psychotic behaviors.
As expected, clients with substance abuse problems showed greater use and reported more problems with alcohol and drugs, compared with those with mental health problems, including: more days intoxicated during the past 3 months (2.5 vs. 0.4), a higher rate of illicit drug use (35% vs. 24%), and higher ASI alcohol and drug scores (0.13 vs. 0.03 and 0.04 vs. 0.02, respectively) (Table 12).
Clients with dual mental health and substance abuse problems fell somewhere between these other two disability groups in the areas of knowing neighbors well, receipt of public assistance, total income, and observed psychotic behavior. Dual problem clients reported a lower level of mental health functioning, and greater alcohol and drug use problems and a higher rate of illicit drug use (44%) than either of the other two psychiatric disability groups, as well as higher quarterly medical/dental, substance abuse treatment, and total health care treatment costs ($5,422 vs. $3,104-$3,869), including both higher inpatient costs ($4,284 vs. $2,471-$3,216), and higher outpatient costs ($1,151 vs. $636-$653) (Table 12).
Younger clients spent more days in jail (1.3 vs. 0.5), received less public assistance ($392 vs. $475) and total income ($474 vs. $571), had a lower level of mental health functioning (39.3 vs. 41.3), were more distressed by psychiatric symptoms (1.46 vs. 1.21), and were more likely to use illicit drugs (40% vs. 29%) than older clients. As one would expect, younger clients showed a higher level of physical functioning (45.9 vs. 42.8), and relatedly, lower medical/dental treatment costs ($1,799 vs. $2,881) than older clients (Table 12).
Compared with Caucasian (non-Hispanic) clients, minority clients were more likely to be housed (83% vs. 80%) and had been housed for a greater number of days (68 vs. 64) during the previous 3 months. Minority clients also had larger social support networks (1.4 vs. 1.3), were more satisfied with life overall (4.7 vs. 4.4), had a higher level of mental health functioning (40.4 vs. 39.3), lower medical/dental and total health care treatment costs ($3,892 vs. $5,391), and, more specifically, lower inpatient care costs ($2,954 vs. $4,543) than Caucasian clients (Table 12).
Clients who had been incarcerated during their lifetimes prior to entering CICH were more likely to be homeless at the time of follow-up assessment, spent more days in jail, were more distressed by psychiatric symptoms, had higher rates of illicit drug use and greater alcohol and drug problems, as well as higher substance abuse treatment costs than clients without an incarceration history (Table 12).
Veterans were more actively involved in the community and had greater public assistance and total incomes than non-veterans, yet were also less likely to be housed at follow-up and were less satisfied with life in general.
Female clients were more likely to be housed than males and were housed for more days, were more satisfied with life in general, and were more likely to receive public assistance. However, female clients participated less in community activities, were less likely to be employed, presented with more psychotic behavior during interviews, had a lower level of mental health functioning, and had higher mental health services treatment costs (Table 12).
Significant differences between sites were found on all 19 service use measures (Table 13), and on 27 of 28 client outcome measures (excepting the average cost of medical and dental treatment) (Table 14).
For example, in New York, relatively few clients had someone managing their money (2% at baseline, rising to 4% at 3, 6, and 9-months to 7% by 12-months) (Table 13). In San Francisco, nearly all clients had money managers (85% at baseline and rising to 97-99% from 3 through 12-months). The proportion of clients having money managers in Los Angeles, Martinez and Philadelphia remained fairly constant around 18%, 24%, and 40%, respectively. Other sites showed rapidly increasing proportions of clients receiving money management, including Chattanooga (from 6% at baseline to 20% at 12-months) and Columbus (from 11% at baseline to 49% at 12-months).
Considering an outcomes example, the average number of days housed at baseline ranged from 5 in New York to 42 in San Francisco (Table 14). The spread between low and high remained relatively high at 3-months, ranging from 49 days in Portland to 85 days in San Francisco. However, beginning at 6 months and continuing to 12-months, the variability across sites on this outcome measure decreased, ranging from 78 days in Los Angeles to 90 days in Ft. Lauderdale at 12-months.
After adjusting for service use measures bivariately associated with each outcome, differences in outcomes across sites remained statistically significant on all but four cost measures: medical/dental, substance abuse treatment, inpatient care, and total healthcare costs, suggesting that differences in clinical outcomes are not accounted for by differences in service use (Table 15).
An analysis of aggregated data pooled across sites and time points showed that after, adjusting for site and service use measures bivariately associated with each outcome, each service use measure was significantly associated with at least one outcome measure.
Service use measures associated with the largest number of outcome measures included participation in alcoholics anonymous or narcotics anonymous (AA/NA) (associated with 11 outcomes), coordination of services, and having a money manager (9 outcomes), along with therapeutic alliance with a mental health/substance abuse provider (8 outcomes). The remaining 11 service use measures were associated with between 3 and 6 outcome measures (Table 16).
The two service use measures associated with the smallest number of client outcome measures were: a) any contact with landlord and b) total number of outpatient visits, which were each associated with only one outcome. Landlord contact was associated with a higher level of mental health functioning, and total number of outpatient visits with decreased total health care costs, indicating a savings in total health care costs of $61 per outpatient visit, on average (Table 16). The number of outpatient mental health visits and the number of different service providers were each associated with only two client outcomes.
Clients who participated more frequently in AA/NA meetings were housed an average of 8 days less per quarter than other clients. and showed significantly higher healthcare costs for all 3 types of care examined as well as for total outpatient costs. Those with more serious substance abuse problems most likely required a higher level of use of healthcare services. These clients also experienced more distress due to psychiatric symptoms. However, those participating in AA/NA were more actively involved in the community, had larger social networks, were more likely to be employed, and were more satisfied with their primary mental health/substance abuse treater than other clients (Table 16, column 9).
Clients assigned to a money manager had more negative outcomes, including: fewer days of housing, more psychotic behaviors, and higher costs in the medical and mental health domains in addition to overall higher inpatient care costs. Clients having a money manager were also more likely to receive public support payments, and consequently, more total income than did clients without a money manager (Table 16, column 14). Presumably assignment of a money manager reflected these circumstances rather than being caused by them.
Clients receiving more coordinated services had better outcomes than others, including having a greater number of days housed and more satisfaction with their housing. Coordination of services was also positively associated with greater satisfaction with life overall and greater trust in one's doctor; with higher levels of mental health functioning and less distress due to psychiatric symptoms; and with decreased rates of illicit drug use and less severe drug problems (Table 16, column 18).
As with coordination of services, the strength of the therapeutic alliance between clients and primary mental health or substance abuse providers was associated with several positive outcomes including greater housing satisfaction and greater satisfaction with life overall. The alliance was also associated with larger social support networks, higher levels of mental health functioning, and with lower rates of drug use and less serious drug problems (Table 16, column 10).
Compared with CICH clients, comparison group subjects were less likely to have experienced 4 or more episodes of homelessness (54% vs. 68%), and less likely to have been primarily unemployed (34% vs. 46%), within the previous 3 years (Table 17). They were also less likely to be Asian/Pacific Islander (0% vs. 3%), and had fewer mental health problems than CICH clients. Comparison group subjects also had fewer mental health and substance abuse diagnoses (2.6 vs. 3.6), with a lower likelihood of having been diagnosed with bipolar disorder (12% vs. 37%) or with anxiety disorder (22% vs. 44%). Differences in access to healthcare service at baseline were mixed, with comparison group subjects reporting higher rates of being uninsured (34% vs. 17%) and having fewer healthcare providers (3.7 vs. 4.7) than CICH clients, while being more likely to have a regular source of healthcare (90% vs. 76%) (Table 17). Comparison group subjects were also more likely than CICH clients to have received vocational rehabilitation services during the previous 3 months (25% vs. 13%), but were less likely to have received housing services (42% vs. 91%) or case management visits in the community (4% vs. 52%) than CICH clients. Finally, comparison group subjects had spent fewer days in the hospital during the previous 3 months than CICH clients (0.6 vs. 2.2), while being somewhat more religious and placing a greater importance on faith in their lives than CICH clients (2.1 vs. 1.9, on a scale from 0-3).
After adjusting for site, the baseline value of the dependent variable, and the 14 baseline covariates described above and presented in Table 17, significant differences between the two groups during the 12-month follow-up period were found on five service use measures and on two outcome measures. Overall, CICH clients were 1) more likely to have health insurance (i.e., were less likely to be uninsured), 2) more likely to have a primary treater for mental health or substance abuse problems, 3) more likely to be visited by a case manager in the community, 4) had a greater number of different healthcare providers, and 5) received a more integrated "package" of housing and supported services than did comparison group subjects (Table 18, "group" column). CICH clients were also housed a greater number of days, were less likely to be homeless at the time of assessment, and had higher public support incomes than comparison group subjects. No other statistically significant differences between groups were found.
Rates of change in the proportions of individuals having a usual treater for both physical health problems and mental health/substance abuse problems increased for CICH clients, while decreasing for comparison subjects. The same was true of the proportion of subjects receiving a more integrated package of housing and supported services, although the declining proportion among comparison group subjects was less than was the case for the two primary treater/provider measures (Table 18, "group*time" column). Both groups showed increasing rates of being visited in the community by a case manager, but the rate of change was greater for CICH clients than for comparison group subjects.
Both groups showed dramatic improvements in housing rates over time, although the rate of improvement among CICH clients was substantially greater. CICH clients' participation in community activities remained largely unchanged, while comparison group subjects' participation declined modestly over time. A gradual, modest decline in the average number of days in jail was found among CICH clients, in contrast to more widely fluctuating decreases and increases in jail time observed from quarter to quarter among comparison group subjects.
Finally, there were no significant differences in total costs or changes in cost over time, although the overall pattern of total cost data suggests that CICH clients began with somewhat higher costs at baseline, but that at 12 months costs were not substantially different. Some differences were found between the groups in rates of change on the measure of quarterly substance abuse treatment costs, but these fluctuations are likely to reflect the relatively small number of clients in the comparison group.
[ Go to Contents ]
The primary purpose of the national evaluation of client outcomes in this evaluation was to monitor both the use of services and client outcomes systematically and uniformly among all 11 sites. The major goals of the program included: 1) rapid placement of chronically homeless individuals in permanent housing, and once placed, helping them to retain (keep) this housing, 2) providing an integrated set of housing and supported services to those enrolled into the program to improve health outcomes, 3) assisting those enrolled in accessing mainstream resources, most notably public assistance income, and 4) obtaining alternative long-term funding to sustain program services beyond the 3-year grant funded by HUD, DHHS and VA.
This report has addressed the first 3 goals of the program mentioned, all of which were successfully addressed. The average number of days housed at each follow-up assessment increased from 18 out of the previous 90 days at baseline to 68 days at 3-months and upwards to 83 days at 12-months. The vast majority of CICH clients were thus placed into housing within 3 months of enrolling into the program, and nearly all of those placed remained housed one year later.
The proportion of core services received also increased from 64% at baseline to 78-81% thereafter, indicating that CICH providers effectively maintained contact with most CICH clients and were able to facilitate their engagement in treatment during the first year following enrollment into the program. In association with these changes, mental health functioning improved while psychological distress decreased steadily and significantly. However, more than one-third of clients reported using illicit drugs at each assessment, and alcohol and drug problems remained largely unchanged over time. It is possible that more time is needed for improvements to emerge in this outcome domain.
Thirdly, the percentage of CICH clients receiving public assistance increased from 70% at baseline to 81-85% thereafter. Mean monthly public assistance income increased steadily from $316 at baseline to $478 one year later, a 50% increase.
The 50% reduction in total average quarterly health care services costs- from $6,832 at baseline to $3,376 at 12-months is encouraging although the lack of a randomly assigned comparison group precludes any causal inference.
Among the six client sub-groups examined, minorities showed consistently more favorable outcomes than non-Hispanic whites. The remaining five sub-groups (defined by disability status, age, legal status, veterans status, and gender) all had a mixture of "better" and "worse" outcomes. Sub-group analyses results were thus variable and do not suggest particularly beneficial effects for specific subgroups.
Comparisons of overall group differences and of rates of change between CICH and comparison clients at the five involved sites provide evidence that CICH increased access to housing, primary providers of both physical health and mental health, community-based case management, and a more integrated "package" of housing and supported services as well as a large number of total service providers than were available to other chronically homeless individuals in these communities.
Perhaps as a result of providing a fuller bundle of services, CICH clients showed substantially greater improvement in the number of days housed, throughout the 12-month follow-up period, without incurring significantly higher treatment costs and without any deleterious clinical outcomes found between the groups. There were no differences in other outcome measures.
The major limitation of this preliminary report is that the comparison group used in examining the effectiveness of the CICH program was not identified through random assignment. Five sites recruited a comparison group, and these clients turned-out to differ from CICH clients on 80 of 290 baseline measures examined. Nevertheless, after adjustment, the housing benefits of CICH emerged quite robustly.
The other (second) major characteristics, albeit not necessarily a limitation, is the considerable variation observed across program sites, including recruitment procedures, target client populations, the use of scattered site vs. congregate (SRO hotel) housing placements, geographic region, etc. While we have statistically adjusted for site differences, such adjustment can only partially reduce variability across sites.
This variability may, in fact, be a strength of the CICH demonstration because it shows that program objectives were achieved across a heterogeneous array of sites. Real-world dissemination of new initiatives invariably takes place in a complex and diverse array of locations.
These preliminary findings suggest that a diverse population of chronically homeless adults with disabling conditions can successfully be housed and can maintain their housing when provided with permanent housing services, intensive case management, and access to primary healthcare and mental health/substance abuse treatment. Data from comparison sites suggest that CICH clients experienced improved housing outcomes at no greater treatment cost.
[ Go to Contents ]
Anderson, L.A. & Dedrick, R.F. (1990). Development of the Trust in Physician Scale: A measure to assess the interpersonal trust in patient-physician relationships. Psychological Reports, 67:1091-1100.
Burt, M.R., Aron, L.Y., Lee, E., & Valente, J. (2001). Helping America's homeless: Emergency shelter or affordable housing? Washington, D.C.: The Urban Institute Press.
Bryk, A.S. & Raudenbush, S.W. (1992). Hierarchical linear models. Newbury Park, CA: Sage Publications Inc.
Center for Mental Health Services. (2001). The CMHS housing initiative. Rockville, MD: Substance Abuse and Mental Health Services Administration.
Culhane, D.P., Metraux, S., & Hadley, T. (2001). The impact of supportive housing for people who are homeless with severe mental illness on the utilization of the public health, corrections, and emergency shelter systems. Washington, D.C.: Fannie Mae Foundation.
Derogatis, L.R. & Spencer, N. (1982). The brief symptom index; Administration, scoring and procedure manual, Baltimore MD, Johns Hopkins.
Dohrenwend, B. (1982). Psychiatric Epidemiology Research Interview (PERI). New York: Columbia University Social Psychiatry Unit.
Heslin, K.C., Andersen, R.M., & Gelberg, L. (2003). Case management and access to services for homeless women. Journal of Health Care for the Poor and Underserved, 14(1):34-51.
Interagency Council on Homelessness (ICH). (2006). URL: http://www.ich.gov
Katz, A.H. (1963). Social adaptation in chronic illness: A study of hemophilia. American Journal of Public Health and the Nation's Health, 53(10):1666.
Kloos, B., Zimmerman, S.O., Scrimenti, K., & Crusto, C. (2002). Landlords as partners for promoting success in supported housing: "It takes more than a lease and a key." Psychiatric Rehabilitation Journal, 25(3):235-244.
Kuhn, R. & Culhane, D. (1998). Applying cluster analysis to test a typology of homelessness by pattern of shelter utilization: Results from the analysis of administrative data. American Journal of Community Psychology, 26(2):207-232.
Lehman A. (1988). A quality of life interview for the chronically mentally ill. Evaluation and Program Planning, 11:51-62.
McLellan, A., Luborsky, L., Woody, G, et al. (1980). An improved diagnostic evaluation instrument for substance abuse patients: the Addiction Severity Index. Journal of Nervous and Mental Disease, 168:26-33.
Monahan, J., Redlich, A.D., Swanson, J.R., Clark, P., Appelbaum, P.S., Petrila, J., Steadman, H.J., Swartz, M., Angell, B., & McNiel, D.E. (2005). Use of leverage to improve adherence to psychiatric treatment in the community. Psychiatric Services, 56(1):37-44.
National Technical Assistance Center (NTAC). (2006). Unpublished summary of field notes taken during visits to CICH sites.
Neale, M.S. & Rosenheck, R.A. (1995). Therapeutic alliance and outcome in a VA intensive case management program. Psychiatric Services, 46(7):719-723.
NOFA. (2003). Notice of funding availability for the Collaborative Initiative to Help End Chronic Homelessness. Federal Register, Vol. 68, No. 17, January 27, 2003.
Robins, J.M. & Finkelstein, D. (2000). Correcting for noncompliance and dependent censoring in an AIDS Clinical trial with inverse probability of censoring weighted (IPCW) Log-Rank tests. Biometrics, 56:776-788.
Robins, J.M. (1993). Information recovery and bias adjustment in proportional hazards regression analysis of randomized trials using surrogate markers. In Proceedings of the Biopharmaceutical Section, American Statistical Association, pp. 24-33, American Statistical Association: Alexandria, VA.
Rosenheck RA, Leslie D, Sindelar J, Miller EA, Lin H, Stroup S, McEvoy J, Davis S, Keefe RSE, Swartz M, Perkins D, Hsiao J, Lieberman JA (2006). Cost-Effectiveness of Second Generation Antipsychotics and Perphenazine in a Randomized Trial of Treatment for Chronic Schizophrenia. American Journal of Psychiatry; 163(12):2080-2089.
SAMHSA. (2003). Blueprint for change: Ending chronic homelessness for persons with serious mental illnesses and co-occurring substance use disorders. DHHS Pub. No. SMA-04-3870, Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration.
Stein, L.I. & Test, M.A. (1980). Alternative to mental hospital treatment: I. Conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry, 37(4):392-397.
Tsemberis, S. & Eisenberg, R.F. (2000). Pathways to housing: supported housing for street-dwelling homeless individuals with psychiatric disabilities. Psychiatric Services, 51(4):487-493.
Tsemberis, S., Rogers, E.S., Rodis, E., Dushuttle, P., & Skryha, V. (2003). Housing satisfaction for persons with psychiatric disabilities. Journal of Community Psychology, 31(6):581-590.
Vaux, A. & Athanassopulou, M. (1987). Social support appraisals and network resources. Journal of Community Psychology, 15:537-556.
Ware, J.E., Kosinski, M., & Keller, S.E.. (1998). How to score the SF-12 physical and mental health summary scales, 3rd edition. Lincoln, Rhode Island: Quality Metric Inc.
[ Go to Contents ]
To obtain a printed copy of this report, send the title and your mailing information to:Human Services Policy, Room 404E
Fax: (202) 690-6562
Top of Page | Contents
Evaluation of the CICH: Project Page
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)
Last updated: 03/11/08