Toward Understanding Homelessness: The 2007 National Symposium on Homelessness Research. Accountability, Cost-Effectiveness, and Program Performance: Progress Since 1998.

Accountability, Cost-Effectiveness, and Program Performance: Progress Since 1998

Dennis P. Culhane, PhD, University of Pennsylvania, Philadelphia, PA
Wayne D. Parker, PhD, The Virginia G. Piper Charitable Trust, Scottsdale, AZ
Barbara Poppe, Columbus Emergency Food and Community Shelter Board, Columbus, OH
Kennen S. Gross, MPH, University of Pennsylvania, Philadelphia, PA
Ezra Sykes, MPA, Massachusetts Housing and Shelter Alliance, Boston, MA

Abstract

The authors summarize the progress made in the past decade toward making homeless assistance programs more accountable to funders, consumers, and the public. They observe that research on the costs of homelessness and cost offsets associated with intervention programs has been limited to people who are homeless with severe mental illness. But this research has raised awareness of the value of this approach, such that dozens of new studies in this area are underway, mostly focused on "chronic homelessness." Less progress has been made in using cost and performance data to systematically assess interventions for families, youth, and transitionally homeless adults. The authors present case studies of promising practices from the State of Arizona and Columbus, Ohio, demonstrating innovative uses of client and program data to measure performance and improve program management toward state policy goals, such as increased housing placement rates, reduced lengths of homelessness, and improved housing stability.

Disclaimer: This paper was developed for the National Symposium on Homelessness Research held on March 1-2, 2007. The Symposium was conducted by Abt Associates Inc. and Policy Research Associates Inc. under contract for the Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services; and the Office of Policy, Development, and Research, U.S. Department of Housing and Urban Development. The paper presents the views and opinions of the respective author(s) and does not necessarily represent the views, positions, and policies of the federal government.

Introduction

This paper summarizes the progress made in making homeless assistance programs more accountable to funders, consumers, and the public since the first national homelessness research symposium in 1998. Some published studies related to the costs and cost offsets of homeless assistance programs are reported, although the published literature on costs remains relatively modest. A recent spate of research on the costs of chronic homelessness has emerged, most of which has yet to make it into the published literature and some of which is not intended for an academic audience. These new studies are playing an instrumental role in local “10-year plans” to address chronic homelessness and are summarized here for the first time. The literature on cost, cost-effectiveness, and program outcomes for populations who are not experiencing chronic homelessness or who do not have a severe mental illness, has experienced even more limited growth. Relatively little progress has been made in identifying systematically the types of interventions that work for families, youth, and transitionally homeless adults. However, some new or newly documented initiatives for families, including innovations in housing stabilization and relocation programs, may soon provide evidence on the cost-effectiveness of these interventions as compared to emergency shelter and transitional housing. The growing number of cities that have implemented management information systems to track the use of homelessness services has also begun to create a more general capacity for better data and more accountability of homeless assistance programs in many communities. Progress in this area is documented here and holds promise that an infrastructure can be established for the more standardized assessment of program performance and for improving local service systems in the near future. Two jurisdictions in particular, the State of Arizona and Columbus, Ohio, have made innovative use of client and program data to measure program performance and to improvement the management of their service systems toward explicit policy objectives. Those efforts are examined as best-practice case studies here.

Background

In their 1998 paper, Making Homelessness Programs Accountable to Consumers, Funders and the Public (Culhane et al., 1999), the authors provide a framework for assessing program outcomes that addresses the information needs of the various constituencies for homelessness services (consumers, funders, and the public). Consumers, it was argued, need data on services received and whether those services meet their perceived needs. Funders require program-level performance data both to demonstrate that programs are delivering the services they are funded to perform, and to compare providers on standardized performance benchmarks. The public needs system-wide performance measures that demonstrate whether the system as a whole is meeting its primary objectives of improving the lives of homeless people and reducing homelessness, and to demonstrate if it is doing so in as efficient and cost-effective a manner as practicable. A variety of studies were cited that illustrated different approaches to these areas and that demonstrated that the measurement of outcomes of interest to these respective audiences was indeed possible. Nevertheless, the empirical evidence on program accountability was quite limited at that time. Little systematic research was occurring on a national scale, aside from a few federally sponsored research demonstrations and the research based on the nationally standardized program data from the U.S. Department of Veterans Affairs. The research was also limited in its timeliness and utility, in that it was frequently geared toward academic audiences, appearing in professional journals years after the completion of the interventions studied. The authors recommended an expansion of standardized data collection tools, including more brief questionnaires that would have program-level and local service system utility. They also argued for a broader, more “systems-wide” view of homeless assistance programs that would examine the relationship of homeless assistance programs to other, mainstream social welfare systems. The authors encouraged the development of automated systems that could track clients longitudinally and that could be used to assess program effectiveness on a more routine basis. To accomplish this, the authors also urged closer collaboration between researchers and practitioners.

Since the publication of that paper, important advances have been made, although much remains to be done. In general, our sense is that the rate of published research on homelessness has slowed since 1998. For our purposes here, we have chosen to focus on the areas where there does seem to be a growing interest and/or literature: analyses of the costs of homelessness and the cost offsets associated with various interventions; and the use of standardized data collection tools and performance reporting in homeless assistance programs. Unfortunately, the literature has grown unevenly with respect to the various subpopulations who experience homelessness. As has been the case for the last 25 years, since 1998 research on homelessness among people who have a severe mental illness (SMI) is more prevalent than research on other populations, perhaps owing to the funding priorities of federal research sponsors such as the National Institute of Mental Health (NIMH) or the Substance Abuse and Mental Health Services Administration (SAMHSA). The bulk of our literature review will, therefore, focus on services and housing for persons with severe mental illness, and the review is far less comprehensive for other persons who are homeless. Although there has been a recent increase in studies of chronic homelessness, beyond people with severe mental illness, this is a relatively new phenomenon; most prior research has not included people who do not have a severe mental illness, even among the chronically homeless — for example, people whose primarily disability is related to substance use — although people without severe mental illness are the majority of people who experience chronic homelessness. Hence, as a result of both the reduced rate of published research since 1998 and its continued focus on persons with mental illness, the available literature is frustratingly narrow in applicability to the homelessness problem overall.

That said, some research on families and youth has begun to inform interventions for these populations, as has literature on adults who are not chronically homeless but who are transitioning from institutional programs (prison, foster care, detoxification). Experimental tests of these interventions are not yet available, and neither are there many detailed cost and utilization studies associated with these subpopulations. Where limited data are available — for example, for families — they will be discussed here.

Despite the limited progress to be found in the research literature, policymakers have made substantial progress in increasing the accountability of homeless assistance programs since 1998. Most significantly, the requirement by Congress in 2000 that communities implement Homelessness Services Management Information Systems (HMIS) has been responsible for a broad and systematic expansion in data collection on both clients and programs. To date, actual research based on these data has been limited, as most communities have had to overcome various technical and human resource challenges during system implementation. However, a growing body of cities have successfully implemented HMIS, and as that pool grows, analyses based on these data will surely find their way into the published literature. The increasing interest in costs and the cost offsets associated with various interventions and subpopulations could also drive more research based on HMIS data. Absent information identifying emergency shelter users from HMIS, it is difficult for communities to track services use across other service systems, let alone within shelter and housing programs. One could argue that the literature on costs and utilization has not grown as quickly as one might have expected since 1998 because of the slow pace with which these information systems have been fully implemented. However, as implementations proceed, more data should yield more analysis and knowledge development, including for populations of homeless people who are not experiencing chronic homelessness.

Until the relatively recent adoption of HMIS, the lack of standardized data collection also restricted the growth of performance measurement and performance-based contracting in the field of homelessness. Performance measurement generally has been limited to fulfilling reporting requirements, such as HUD’s Annual Performance Report (APR). While such performance reports enable HUD to conform to the Government Performance and Results Act of 1993 (GPRA), they have not generally served as practical tools for program improvement and management at a local level (with some exceptions, see the Columbus case study below). A few communities have set up other reporting systems that serve as the basis for performance-driven planning and contracting. Such reporting and feedback processes are necessary for guiding service systems toward a desired set of policy objectives. The HMIS infrastructure could be used as the basis for such systems in the near future. Innovators in this area, such as Arizona and Columbus, Ohio, have shown how quality program-level information can be used for such purposes.

Culhane et al. (1999), in their “accountability” paper, established a framework for considering comprehensively how to measure the accountability of homeless assistance programs to funders, consumers, and the public. However, progress since then has been mixed, and only a subset of programs (mostly a particular type and intensity of supportive housing) and a subset of the homeless population (people with severe mental illness, and some people who are chronically homeless) have been studied to any significant degree since the last symposium. We still know very little about the accountability of other service interventions and the costs associated with homelessness among several important population groups. What follows here is an assessment of what we do know, how the field is working systematically to improve our knowledge base, and some examples of how communities can integrate accountability and management standards into their practices so that further advances can be made across a broader spectrum of programs and populations in the near future.

Literature Review: The Cost of Homelessness and Its Alternatives

Some advances in knowledge about the effectiveness of homeless assistance programs and the costs of intervening — or not — have been achieved since the publication of the Culhane et al. (1999) accountability paper. A growing interest in demonstrating the cost and cost offsets associated with programs targeting people who experience “chronic” homelessness has led to a recent growth of research in this area. Not all of this research has yet made it into academic journals; indeed, some of it has been intended foremost for a policy audience. Most commonly, cost and cost offset studies are based on aggregations of services utilization data, and costs are imputed based on unit costs derived from budget documents or reimbursement rates. More formal cost-effectiveness and cost-benefit studies have seldom been conducted.

Culhane et al. (1999) provide a framework for comparing and describing the differences among the various cost-based evaluation research methodologies, and readers are referred to that document for more complete descriptions of their approaches. In this paper, we focus on the lessons from the published research on the costs and cost offsets of homeless assistance programs, and we examine and discuss the implications of the rapidly emerging literature on chronic homelessness.

Studies of the Costs of Homelessness

Researchers and others have been interested in documenting the costs of homelessness because it is believed that demonstrating high costs will inspire investments in alternative housing and services. Indeed, as will be discussed in the next section, several investments in alternative housing and service models have included evaluations to examine the degree to which the costs of such interventions are offset by reducing the excess costs associated with homelessness. But is homelessness costly? Before reviewing the literature on interventions, we examine the assumption that notintervening carries some significant costs. Although the identification of such costs may not reflect on the accountability or effectiveness of homeless assistance programs per se, they may shed light on those social welfare systems that should be more accountable, or whose collective costs for accommodating homelessness might compel society to be more accountable.

Studies have found high rates of emergency room use and high rates of hospitalization for mental health and substance abuse problems among adults who are homeless, particularly in comparison to other low-income, but housed populations. Kushel et al. (2002) found that unstable housing and homelessness were associated with more emergency room use than was marginal housing. Kuno et al. (2000) also found that homelessness among a sample of people with severe mental illness was associated with greater inpatient admissions and longer hospital stays, as compared to a non-homeless comparison group. Salit et al. (1998) found that homeless adults in public hospitals in New York City stayed on average 36 percent longer than other patients, controlling for differences in demographics and diagnoses. While this study did not distinguish types of homeless persons or degrees of services use, it did highlight the nature of the health problems of persons who are homeless and hospitalized, with 80 percent of the primary or secondary diagnoses including substance abuse or mental illness. 

In his review of the cost-effectiveness literature on homeless assistance programs, Rosenheck (2000) observes that, while service use may be greater among adults who are homeless than those who are not, heavy service use is relatively uncommon, even among people with severe mental illness. A lack of health insurance and a lack of access to primary care are typical for people who are homeless, so it is quite possible that many people either go untreated or avoid care altogether. Rosenheck cites data from two programs, the Homeless Chronically Mentally Ill Program (HCMIP) and the Access to Community Care and Effective Supportive Services Program (ACCESS), in which average annual inpatient costs for participants were $7,905 and $8,346 respectively (1996 dollars). However, at the 90^th percentile, average costs reached $32,605 and $25,010, respectively, leading him to conclude that only the most costly 10 percent of the people who are homeless and have mental illness are likely to have such excess costs as to be able to demonstrate a sufficient offset for the costs of the interventions under study. It is worth noting that enrollees in these programs had to have a serious mental illness, which occurs in about 20–25 percent of the adult homeless population (Lehman & Cordray, 1993).

Other research has supported the conclusion that a costly subpopulation of homeless people does exist and appears to be quite distinct from the single adult homeless population in general. Cluster analyses based on shelter utilization data in New York City and Philadelphia has identified a “chronic homeless” population that stays in shelter for long periods of time but represents only 10 percent of adult shelter users overall (Kuhn & Culhane, 1998). Because of their heavy utilization, they account for 50 percent of the total number of shelter nights or of the total annual public expenditures for shelter. Nearly all of the chronic shelter users have a treatment history of severe mental illness or substance abuse, or a physical disability. The average shelter cost for the chronic shelter user population was $6,600 in Philadelphia and $20,400 in New York City (2006 dollars). In neither city was it possible at the time of the study to track episodes of street homelessness, nor street outreach contacts, so costs and total days and episodes of homelessness are underestimated. For the chronic shelter users, even ignoring their use of other service systems, annual shelter costs may equal or exceed the costs of providing rental assistance in many housing markets.

Part of the challenge in identifying costs associated with people who are homeless is obtaining sufficient data to document those costs. Consumer self-report poses reliability issues, so often researchers have relied on administrative data to measure service utilization and costs. As will be discussed later, administrative data come with their own challenges, especially limited accessibility. But when available, administrative data can provide detailed information on diagnoses or charges (in the case of criminal justice) and on admission and discharge dates, all of which can be used to infer costs. However, every study is limited by the administrative data it can include (or does not include). For example, a study that includes only VA hospitalization data or Medicaid data will miss state psychiatric facility inpatient days, shelter days, jail and prison stays, or uncompensated care provided in public or private hospitals. The inclusion or exclusion of particular systems can have significant impacts on the assessment of overall costs.

A multi-system study of approximately 5,000 people with severe mental illness who were homeless in New York City found that the average annual service utilization costs were $40,500 per person (1999 dollars) (Culhane, Metraux & Hadley, 2002). This per person average is much higher than those reported from the HCMIP and ACCESS programs discussed above, which were closer to $8,000. Part of the difference may be attributable to the inclusion of data from more systems of care in the New York study. For example, had the authors added only Medicaid inpatient days, the cost would have been $11,500.

Even considering the inclusion of multiple systems, the inpatient mental health costs for the group studied in New York are still markedly higher than for the national sample, and are more in line with the heaviest service users (90^th percentile) identified by Rosenheck (2000). Because this study did not involve random selection but was based on enrollees in a housing program, it is possible that the sample was biased to include more costly service users or people who were engaged in intensive services prior to enrollment. Part of the difference may also reflect regional variations in access to care, in that New York has relatively generous public health and mental health systems compared to other regions in the US, as well as the nation’s only court-enforced “right to shelter.”

The possibility of regional factors is further suggested by results from a multi-system utilization study in Houston (Sullivan et al., n.d.). The Houston study found that homeless people with severe mental illness used an average of $3,700 per year (1996 dollars) in health, mental health and criminal justice services (police and courts, not jail/prison), and that people who were homeless and without SMI used an average of approximately $2,700. Neither estimate includes shelter or outreach costs. In stark contrast to the New York results, these utilization costs for people with SMI are less than half of the national average from the VA and ACCESS programs. Because of its sample design, the Houston sample is likely to be broadly representative of a cross-section of adults who are homeless, as in the VA and ACCESS programs, rather than of intensive service users as may have been the case in the New York study.

Moreover, regional factors in access to care also likely play a role, particularly limited access to mental health services for people who have a severe mental illness. Texas state mental health agency expenditures equaled $37 per capita in 1997, as compared to $116 per capita in New York State, a threefold difference (Lutterman & Hogan, 2000). Access to mental health services may be even more constrained for people who are homeless, as the study results show that the comparison sample of people with SMI who were housed used five times as many mental health services as the people with SMI who were homeless. Regardless, the results indicate that in some regions of the U.S., limited access to services for people who are homeless may result in underutilization of services, and therefore lower costs than for other low-income persons. (This would inherently limit the potential for demonstrating cost offsets associated with alternative program placement, as there are relatively few costs to reduce in the first place.)

Although research on the costs and cost-effectiveness of homeless assistance programs (and homelessness in general) appears to have slowed since the 1998 symposium, there has been a recent resurgence in interest in identifying the costs associated with homelessness, and, in particular, chronic homelessness. Since 2000, Congress has required that 30 percent of McKinney-Vento spending be reserved for permanent housing, and HUD has further required that one third of this set-aside be used for projects that serve a population that includes at least 70 percent persons who are chronically homeless. The U.S. Interagency Council on Homelessness (ICH) has similarly focused local and state officials and planners on identifying people who are chronically homeless through local and state 10-year plans to end (chronic) homelessness. Many of these 10-year plans require communities to identify the resources to pay for alternative housing interventions, and this has led a number of them to conduct or sponsor “cost studies” as the basis for garnering political will for their cause.  

The U.S. Interagency Council on Homelessness has recently identified 14 such unpublished “cost studies,” including 11 that are complete and 3 that are ongoing. As shown in the ICH summary of these cost offset studies (see Appendix exhibit), the studies have taken a variety of forms, with some conducted by academic researchers and others by planners. Some were inspired by an article in the New Yorker by Malcolm Gladwell (2006) entitled “Million Dollar Murray,” which summarized the results of the Reno study. Because most of these studies have not been published, many details about the sources of data and about the exact nature of the samples are not precisely known. Nearly all have involved tracking individuals through various administrative data sources, and a handful have involved only aggregations of costs attributed to homeless people by various systems of care.

A general observation about these projects is that they are mostly based on convenience samples, with a few exceptions: one study includes a data match between all HMIS records from Richmond, VA, and a statewide psychiatric inpatient database for the entire state of Virginia; another from Durham, NC, of “verified chronically homeless individuals” required that people be identified as chronically homeless by at least two independent service providers, and appears to include the universe of such persons served by the participating providers. Other projects that used convenience samples, especially those specifically intended to identify the high costs of certain people who are chronically homeless, cannot be generalized to the adult homeless or the chronic homeless population overall. Given a distribution of costs, some subset of persons will have very high costs.

The studies also have varying data sources and time frames for measuring costs, which limits their comparability. Consequently, the results are highly variable and include a broad range of costs per person, from $5,360 per person per year (incarcerated homeless only, and their jail costs only, in Louisville, KY) to $133,333 per person per year (public inebriates only from San Diego, based on EMS, hospitalizations, and police charges). The studies with aggregate results (not based on client-level tracking) indicate that homeless persons have a significant impact on hospitals and other emergency services in total, although we do not know the proportion of total expenditures in these institutions that they represent, nor the number of unique individuals to which these costs can be attributed.

While these studies have limitations and their findings may be regarded as primarily illustrative from a social science standard, they are playing an instrumental role in local policy discussions. In many cities, documentation of such high costs associated with a subset of homeless people, however unrepresentative, is a powerful means of demonstrating the impact of chronic homelessness on society and garnering political momentum around local plans to address it. On that measure, these studies may be even more effective than more polished academic research, having a local basis, involving the participation of local institutions, demonstrating the impact on those local institutions, and often involving known homeless persons in the community. The U.S. Interagency Council on Homelessness has also encouraged communities to identify the most expensive persons, because, regardless of their representativeness, they are real people who can be housed, and likely with significant reductions in costs because they are such high service users. From this perspective, the issue of representativeness is moot until the pool from which to draw people who are likely high-cost service users dwindles appreciably. At that point, modeling costs and cost offsets for the larger population of persons would require a different sample definition.

Finally, it is worth noting that, while some of these projects include academic researchers, further participation by academic partners could bring more value to these efforts. This is an area where federal resources could help to bring some formalization and standards to the research, and, in so doing, could greatly expand the knowledge base.

Some Observations on the Cost of Homelessness Among Families

Homeless families are relatively understudied when it comes to research on the costs of homelessness. A recent study of family shelter utilization patterns and costs has identified a long-stay population that is roughly analogous to the “chronic” shelter stayers identified in the single adult literature (Culhane et al., in press). Most of these families do not have disabilities or other intensive service histories; however, as is the case for single adults without accompanying children, long-stay families represent a minority of the families sheltered (20 percent) but they account for half of the shelter system costs. And these shelter costs alone are quite remarkable. The study included four US jurisdictions, and found that the average cost for the long-stay families ranged from $27,000 to $55,000 per family. These resources are the equivalent of four or more years of a permanent housing subsidy, or they could provide four or more families with a rental subsidy for a year. The prospect of such cost-efficiencies and of supporting more normalized living environments for a larger population suggests that future research is needed on cost-effective alternatives to long shelter stays for families. Furthermore, unlike the research on adult individuals experiencing chronic homelessness, no research to date has looked at the collateral impact of homelessness among families on other service systems such as child welfare, health, mental health, or education. Other research has also shown that homelessness can be costly to the child welfare system. In particular, a recent study compared the cost of juvenile detention and residential treatment for youth to the costs of a housing subsidy, noting that stable housing costs nearly a tenth of institutional placement (Van Leeuwan, 2004). These are areas deserving further investigation, as are the costs and utilization patterns of other non-chronic or non-SMI populations, including transitionally homeless adults and adults with substance abuse problems.

Studies of Interventions

Studies of the costs of homelessness do not assess directly the accountability or effectiveness of homeless assistance programs. They have been reviewed because they are a means of assessing whether other social welfare systems, policymakers, or society at large, should be accountable for the fact that homelessness can have potentially negative impacts (costs) to society if insufficiently addressed. To assess further the effectiveness of interventions designed to ameliorate homelessness, the federal government and others have funded research demonstration projects and other experimental and quasi-experimental studies. Rosenheck (2000) and Dickey (2000) published separate reviews of this literature, with an emphasis on the cost-effectiveness of programs. In both cases, the authors distinguish studies of outreach programs, case management and other service interventions, and specialized housing programs. Their reviews will be summarized here by these categories. Although most of the studies reviewed were published before 1998 (e.g., five of the eight studies reviewed by Rosenheck), so little has been published overall that they are included again here, along with the handful of studies published subsequent to these reviews. The growing interest in addressing chronic homelessness has also led to a spate of recent, but as yet unpublished efforts to assess the cost offsets of housing and other service programs; some preliminary results from these studies will be considered here as well.

Outreach

Very little has been published about outreach programs in general, so the literature on costs and cost-effectiveness is consequently slim. An experimental study in New York (Shern et al., 2000) evaluated an outreach program for people living on the streets that included engagement, low-demand shelter/drop-in services, respite housing, and community rehabilitation. The enrolled clients had better outcomes than the control group, including a 54 percent reduction in nights sleeping on the street. However, because the enrolled clients were engaged in more services through the program, and through this engagement were more likely to be hospitalized, Rosenheck concludes that the intervention resulted in increased costs.

Rosenheck also cites the experience of the ACCESS program with street outreach: clients contacted through street outreach had significantly improved housing and clinical outcomes. However, because of the high costs of engagement and enrollment, Rosenheck argues that the outreach likely increased program costs substantially. Only one in five screened candidates entered the program, and engagement took twice as long for the people recruited through street outreach as for those recruited from shelters or other programs. The review does not speculate on the potential costs of outreach per case, nor does it report on how service utilization patterns may have changed as a result of enrollment (such as inpatient or emergency room costs, police contacts, or emergency medical transport charges). However, a more recent evaluation study reported reduced inpatient stays associated with enrollment in the ACCESS program (Rothbard et al., 2004).

Finally, Rosenheck also cites a VA study that tracked outreach and health care costs for people who were homeless and had a severe mental illness. Results showed that costs increased by 35 percent in the year after entry, including an increase of $855 in health care costs and a total of $2,285 after combining case management and residential treatment costs. He concludes that outreach can be costly, when it is effective, but notes, “This is not surprising, since the very reason for conducting outreach is to enhance access to services for the underserved” (p. 1565).¹

Case Management and Other Service Interventions

Two experimental studies from St. Louis (Wolff et al., 1997) and Baltimore (Lehman et al., 1999) examined the cost offsets associated with providing case management services to people who are homeless and have a severe mental illness. In the St. Louis study, the authors found comparable housing outcomes when comparing two assertive community treatment (ACT) models to a broker case management model (assessment and referral), but improved clinical outcomes for the ACT groups. The ACT models cost approximately $9,000 more than brokered case management, but those costs were offset by reductions in inpatient use, making them effectively cost-neutral. Similar results were found in the Baltimore study. People enrolled in the ACT model had improved clinical and housing outcomes compared to standard care. ACT services cost about $8,244 more, but those costs were again offset by reductions in inpatient service use. 

In his review, Rosenheck (2000) questions the generalizability of these findings. As noted in the previous section, Rosenheck compared the average inpatient costs for large national samples of homeless people with mental illness from the VA Chronically Mentally Ill Homeless Assistance Program and the ACCESS program to the samples from the St. Louis and Baltimore studies. The comparison shows that average utilization in both the St. Louis and Baltimore studies is much greater than in the larger and, presumably, more representative national samples. Rosenheck concludes that, while the experimental nature of the local studies gives them high internal validity, they likely had a sample bias by selecting more costly users of services for participation in the study. Cost neutrality for such an intervention would presumably have been much more difficult to demonstrate in the national samples, as average inpatient costs are only approximately $8,000 to begin with. Rosenheck concludes that only in the higher cost groups would cost offsets likely be achieved.

The other service intervention that Rosenheck and Dickey reviewed for its potential cost-effectiveness is the Critical Time Intervention (CTI) experiment in New York City (Susser et al., 1997; Jones et al., 2003). The critical time intervention involved providing support services for a nine-month period to people who were homeless with a mental illness as they transitioned from shelter to housing. The support services involved strengthening ties to services, family, and friends, and providing people with practical and emotional support. The control condition was also transitioning from shelter to housing, and received “usual services” such as referrals to mental health and rehabilitation programs. The CTI group had better housing and clinical outcomes up to 18 months after enrollment in the program. However, the CTI group had more hospital days (3.8), emergency room visits (.3), outpatient visits (12.8), and day program visits (4.4) than the control group, leading Rosenheck to conclude that the intervention likely increased costs (a cost analysis was not part of the original study). This assessment does not include consideration of the shelter and outreach costs associated with increased homelessness (an additional 60 days on average) or related criminal justice system costs for the control group.

In a recent reanalysis of the data from the CTI study (Lennon et al., 2005), the authors were able to distinguish different subgroups within each condition and observed that the experimental condition likely prevented chronic homelessness from occurring and was an added, but as yet unobserved, benefit of the intervention. The experimental and control groups had three similar groups with regard to their pattern of recurrent homelessness, with the exception of a chronic homelessness pattern that emerged in only the control group. This group, while relatively small (8.3 percent), returned to homelessness quickly after their initial exit and remained homeless for the duration of the study. No such group was observed in the CTI group, leading the authors to conclude that CTI was associated with preventing chronic homelessness.

A recent study of an intervention for people with serial inebriation found significant reductions in emergency medical services use among adults, many of whom were homeless (Dunford et al., 2006). Judges offered residential treatment for the duration of custody as part of an experimental intervention to address “serial inebriates.” Of the 156 people who accepted the residential treatment option (58 percent of those offered), costs declined by $5,642 one year after enrollment. The authors were not able to examine costs beyond EMT services and data from two participating hospitals, so the reductions are likely to be underestimates. The authors do not provide an estimate of the costs of the intervention, but given that the intervention included residential treatment, they are likely to be greater than the offsetting costs from the service reductions. Because the study was part of a formal court-administered program, random assignment was not possible. The study data indicate that people who accepted residential treatment had higher costs prior to enrollment than the people who refused the treatment option, a selection bias in the treated sample that would favor greater cost offsets.

Supportive Housing

Published studies of various housing interventions have demonstrated that housing does indeed resolve homelessness, though to varying degrees depending on the nature of the intervention, and with varying cost offsets. An experimental study in Boston (Dickey et al., 1997) of people who were homeless with severe mental illness compared people who were placed in an “evolving consumer” group living arrangement with similar persons placed in independent subsidized housing. The group living model had staff supervision, which was expected to decline over time as consumers developed a mutually supportive community. Housing outcomes were positive in both conditions, with retention rates at 18 months of 83 percent in independent housing and 92 percent in the group settings. No changes in clinical or functional outcomes were associated with either type of placement. The cost of the group living condition was much greater than the cost of the independent living condition, mostly attributable to the staffing costs. There was not a “no housing” condition in the study, so it is not possible to estimate the degree to which housing placement in either condition was associated with reduced inpatient days or other services use. Rosenheck notes that inpatient services use at baseline in both conditions was modest, more in line with the national samples. This would suggest that the intervention included a broadly representative sample of people who were homeless with mental illness and did not target heavy service users. Thus, it is unlikely that an offsetting cost effect would have been found.

Several studies have included study designs that enable some inferences about the impact of housing on services utilization and costs. A quasi-experimental study (Culhane, Metraux, & Hadley, 2002) with a pre/post design and a matched control group evaluated the New York-New York Agreement, a joint state- and city-funded initiative to develop supportive housing for people who were homeless and had a severe mental illness in New York City. The study analyzed administrative records from seven service systems to estimate the impact of supportive housing placement on services utilization for two years post placement as compared to two years prior to placement. The study found that supportive housing placement was associated with declines in hospitalizations, incarcerations, and shelter stays. Ninety-five percent of the costs of the supportive housing were offset by service reductions ($17,200 per unit per year), resulting in an estimated net annual cost of the supportive housing programs of approximately $1,000 per unit per year. While an advantage of the study was its inclusion of multiple systems to measure impacts on services use and costs, it did not include all potential costs, including police and court costs, emergency medical transport, and emergency room costs. The primary limitation of the study is that it did not involve randomization. It is possible that there was sample selection bias, and that people were selected for housing only if they were sufficiently stable or had received sufficient treatment before program entry. Indeed, as noted in previously, the pre-intervention average cost of $40,500 per person per year suggests that a heavy service-using population was targeted for the intervention and that such high utilization made it possible to achieve such a high degree of offsetting costs.

Rosenheck, Kasprow, and Frisma (2003) conducted an experimental study of a housing intervention for veterans who were homeless and had a severe mental illness. The housing condition included a Section 8 housing voucher coupled with intensive case management. It was compared to a condition of intensive case management alone and a standard care condition. The results estimated potential cost offsets through administrative data from the Veterans Administration that track health services use and through participant self-report in a quarterly interview for non-VA health and other services use (including shelter and jail stays). The results found generally positive housing outcomes regardless of the study condition, including standard care. The housing voucher and intensive case management condition was associated with greater cost offsets, and the intervention had a net annual cost of approximately $2,000 per unit per year. According to the authors, a limitation of the study is that attrition was high in the “standard care” condition. It is possible that persons with recurrent homelessness or in other unstable circumstances were differentially lost to follow-up, biasing the sample in the standard care condition to more favorable outcomes, and thereby lessening the observed differences with the experimental conditions. In any case, although the study was based in part on self-report and included randomization, the findings are roughly consistent with the $1,000 net cost found in the NY-NY evaluation.

A supportive housing intervention in San Francisco (Martinez & Burt, 2006) tracked people who were homeless and who had at least two qualifying conditions (an axis I or II mental disorder, a substance use disorder, or HIV/AIDS). A list of eligible persons was generated through a solicitation of applicants recruited at shelter and feeding programs. Just over 200 people were randomly selected from the list to enter one of two supportive housing programs. Their health service use at the San Francisco General Hospital for two years prior to program entry was compared with service use for two years following placement. A small control group from the waiting list was also selected for comparison purposes. The results show that housing placement is associated with declines in emergency room use, hospitalizations, and inpatient days. The authors estimate that the service reductions offset approximately $1,300 of the cost of the intervention, or 10 percent. The authors attribute the comparatively small cost offsets to their limited access to administrative records, having included only visits to the San Francisco General Hospital, and not including other health or social welfare systems. 

Schumacher, et al. (2002) focused only on people with substance abuse disorders (particularly addiction to crack cocaine), rather than mental illness, and randomly assigned people to either abstinence- and work-contingent housing and treatment or day treatment alone. The results show that the enhanced services model (housing, work, and treatment) cost more per person (approximately $7,700 versus $3,300 per year). The authors did not investigate any potential cost offsets from other service systems that might be associated with the enrollment in the study, but it is possible that shelter, criminal justice, and other health system costs could have reduced the net cost of the intervention. Given the modest success of the program (abstinence was higher at 6 months in the enhanced condition but there was no significant difference at 12 months) and given the modest cost, the authors argue that investments in programs providing housing and treatment to adults who are homeless with addictions are comparable in net cost to other common social interventions of comparable value.

A quasi-experimental study by Clark and Rich (2003) compared people who were homeless with a severe mental illness and who were placed in a comprehensive housing program, including rental subsidies, support services, and case management, to similar people who were placed in a case management-only condition. The results again indicated that housing outcomes were positive irrespective of the study condition. However, persons who had high psychiatric symptom severity and high substance abuse had better outcomes in the comprehensive housing than in the case management alone condition. Although the authors did not include a cost component in their analysis, they conclude that “the effectiveness, and ultimately the cost, of homelessness services can be improved by matching the type of service to the consumer’s level of psychiatric impairment and substance use, rather than by treating mentally ill homeless persons as a homogeneous group” (p. 78).

The fact that several of the studies reviewed here have found positive housing outcomes over time, irrespective of treatment conditions, reinforces the idea that narrower program targeting might be considered as a means of improving the efficiency and effectiveness of programs, as suggested by Clark and Rich (2003). As has been previously noted, most people who experience homelessness, including most people who have a severe mental illness, exit homelessness quickly and do not return within three years; most do so without formal exit support from the homelessness service system (Kuhn & Culhane, 1998). The subpopulation with histories of heavy services use, who experience chronic homelessness or who are otherwise unable to exit homelessness without added supports, should be considered the priority target of the more costly interventions. They not only are the people most likely to need them, but they also are the group for whom the opportunity for demonstrating cost-effectiveness is much greater.

Finally, as with “cost studies,” there has been a recent surge in interest in cost offset studies, particularly for interventions targeting people who are chronically homeless. Nineteen such studies have recently been identified by the U.S. Interagency Council on Homelessness (ICH) — more than have been conducted in the entire period preceding 2003. The Appendix exhibit provides ICH’s summary of 14 studies. The studies are being led by a variety of people, including consultants, academics, planners, and advocates. The interventions are primarily permanent supportive housing programs, some of which include an ACT component. The samples are typically enrollees in the various interventions. They do not appear to have been randomly assigned, so, the opportunity for selection bias exists. Because the people targeted by these projects have experienced chronic homelessness, they are likely to be relatively higher service users, as compared to the homeless population in general. Therefore, they are among the populations for whom significant cost offsets are likely to be achieved following a housing intervention. Most of the studies do not include comparison groups and appear to be pre-post designs. The studies vary in the degree to which they included multiple service systems and in the comprehensiveness of their data, but most have included at least some health services data and some criminal justice systems data. Of course, data coverage determines the ability to find costs as well as cost offsets, and variability in coverage and in other study design issues make the studies not truly comparable.

Results of the completed studies indicate substantial reductions in services use associated with both the ACT and supportive housing interventions (half of the studies are ongoing and do not yet have results). Perhaps not surprisingly, the largest study, based on nearly 5,000 formerly homeless people with mental illness in California, reports the most modest cost offset, at $5,614 in hospitalization, incarceration and emergency room costs. The size of the sample suggests that the intervention did not target people who were chronically homeless, but may have been more broadly representative of people with mental illness who experienced some type of homelessness. Several of the studies report annual cost reductions per person in the range of $13,000–$18,000, which would be roughly consistent with cost offset found in the NY-NY evaluation. Intervention costs are not shown in the Appendix exhibit summarizing the studies, so it is not possible to discern here the degree to which these service reductions offset the costs of the interventions. But such average reductions would appear to be fairly substantial, comparable to the costs of supportive housing in many jurisdictions.

The amount of interest and activity in this area suggests that there is a great appetite for research of this nature. The 10-year planning processes and the resulting housing efforts targeting people who are chronically homeless have no doubt inspired communities to evaluate their progress. But this is an area where federal leadership and support could make a valuable contribution, both by helping to establish standards and comparability in the research and by providing funding that would engage academic researchers as partners with local planners and implementers. The use of administrative records and the lack of randomization would seem to preclude NIMH as a source of funding for these efforts. Many of these communities are seeking more timely feedback than is likely in most prospective, longitudinal studies based on primary data collection. While research of that nature continues to be needed, mechanisms are also needed for supporting qualified research based on administrative databases and research that is more evaluation than experimental in nature.

Prevention

In her review, Dickey (2000) also addresses the issue of homelessness prevention, and the need for cost-effectiveness and evaluation research in this area. She includes the Critical Time Intervention study from New York in that category, Otherwise, this is an area where little formal research or experimentation has been done. Lindblom (1991) and Shinn, Baumohl, and Hopper (2001) have written very thoughtful papers on the subject and helped to distinguish program types. Interventions that are based in the community and work with “at risk” households are probably the least likely to be able to demonstrate an impact, because it is not possible to know who would have become homeless had an intervention not been provided. An evaluation of a neighborhood-based homelessness prevention intervention in Philadelphia found that, while few of the people assisted became homeless, there was no net impact on rates of shelter admission from the areas served (Wong et al., 1999). It is likely that greater potential impact on shelter use can be achieved by targeting people who recently became homeless with interventions designed to prevent continuing or recurrent homelessness. By targeting newly homeless people the intervention can more directly assist households whose risk of homelessness is known (they are in shelter), and the effect on continued or repeat shelter stays is more immediately realizable. Moreover, if client assessments can carefully profile people and match them with the appropriate type and amount of the intervention(s), they may also be more likely to be able to demonstrate cost-effectiveness. Caton et al. (2005) suggest that the results of their study of long-term homelessness can be used to identify people at greatest risk of chronic homelessness early on in their homelessness experience, and that they can be targeted with various interventions to prevent chronic homelessness. Similar assessment tools and service matching strategies are needed for families.

Practice and research on homeless families may be beginning to show some promise in the area of prevention. The National Alliance to End Homelessness (2006) has recently documented the experiences of six jurisdictions that are systematically diverting families from shelter or providing relocation assistance as an alternative to shelter stays or unnecessarily long shelter stays. The anecdotal experience of these jurisdictions suggests that many families can be successfully prevented from having to stay in shelter for more than 30 days by providing modest relocation grants or time-limited rental subsidies. As noted previously, recent research on typologies of families’ shelter stay patterns has suggested that long shelter stays are not associated with personal barriers of families, but with program and policy factors that promote long stays (Culhane et al., in press). The costliness of these stays suggests that resources currently being spent to provide long-term shelter² could be reallocated to a prevention and rapid relocation purpose and could serve families in more normalized environments and in a more cost efficient manner. Future research and demonstration projects could investigate the various packages of housing assistance and services that can maximally achieve independent living among families and in the most cost-efficient manner. New initiatives under consideration in Massachusetts, New York, and Philadelphia may herald a new period of experimentation in an area that has been historically understudied.

Conclusion

Research on the cost and cost-effectiveness of homeless assistance programs indicates that costly interventions are not likely to be cost-effective, except for the most costly clients. Generally positive housing and homelessness outcomes frequently found in studies of people with severe mental illness, irrespective of their being in the intervention or control groups of various demonstration projects, further suggest the importance and value of targeting costly programs to people experiencing (or likely to experience) chronic homelessness, or to people who are heavy service users of non-homeless assistance programs. While some people who are chronically homeless may not be heavy users of non-homelessness services, it is likely that inclusion of heavy and modest service users together can still achieve average outcomes that demonstrate substantial cost offsets, if not cost neutrality for the more costly programs. This area deserves more careful study. Unfortunately, it must also be noted that establishing “cost offsets” doesn’t actually mean that savings that accrue in one agency are then transferred to another agency for this purpose. Inter-agency transfers are the exception, and some communities that have sought to use the “cost offset” arguments to advance new program development have learned that demonstrating cost offsets doesn’t mean that anticipated savings will be advanced for this purpose, particularly across agency boundaries.

Unfortunately, much of the published literature is limited almost exclusively to people who have severe mental illness. The absence of research on other important populations is striking. People with severe mental illness are a minority of the homeless, including a minority of the people who are chronically homeless, among whom they comprise approximately 30 percent (Kuhn & Culhane, 1998; Maguire et al., 2005). Because they may also be among the more expensive service users, given their high rates of hospitalization, studies of people with mental illness are likely to overstate the costs of homelessness for this population relative to people without mental illness. People with primary substance abuse problems, who account for a majority of the chronic homeless population, may use fewer services because they have less access to insurance (low-income people with substance abuse issues must have a co-occurring disabling condition in order to qualify for Supplemental Security Income (SSI), hence Medicaid eligibility), and because managed care has severely limited inpatient and residential treatment for substance abuse. This only highlights the importance of including emergency room use, police and jail records, and emergency medical transport data in cost and cost-effectiveness studies, as people with substance use disorders are more likely to use these services than the inpatient services typically tracked in mental health research.

Even after including these data, it may be the case that some of the people who are homeless and who do not have a severe mental illness (and even some who do have a severe mental illness) may not reach the level of costliness that would inspire major investments in new housing and service interventions. On the other hand, it is also likely to be the case that many of the people who are not severely mentally ill or who are not heavy service users are also likely to need less costly interventions. This is particularly noteworthy given that supportive services costs are typically the largest cost component in most supportive housing projects. Future research (including research demonstration projects) could focus on people who are homeless without a mental illness, including people who have substance use disorders and people who are not chronically homeless, as there has been so little research on the service histories of these very important subpopulations. Moreover, most of the interventions which have been tested, and whose costs are known, are based on people with severe mental illness and have examined only fairly costly interventions. Future research could test interventions that are more modest in cost as well, and which address the issue of identifying the least costly interventions that are necessary to achieve positive housing outcomes.

As Rosenheck (2000) and Dickey (2000) both caution, one should not assume that cost-effectiveness can or should be demonstrated in all interventions intended to reduce or end homelessness, or for all people the programs may serve. Being able to demonstrate cost-effectiveness or offsetting cost reductions can certainly help make the case to policymakers and the public that effective interventions can and should be expanded. However, cost-offset and cost effectiveness studies can also be tricky, particularly with regard to determining how to allocate costs that precede the intervention or which may be part of the intervention. For example, is case management associated with street outreach a cost of homelessness, or part of the intervention that gets someone out of homelessness? Where to allocate such costs will significantly impact any results as to the “cost offsets” associated with the intervention. Aside from this and other methodological concerns, reducing homelessness has other less easily quantifiable benefits as well. Rosenheck and Dickey suggest that other methods be explored to assess both the less tangible benefits of reduced homelessness and the value that the public places on achievement of such an outcome. As Rosenheck also notes, arguments can be made that housing is a social necessity in an advanced society such as ours, and that we have a collective responsibility for making sure that resources are distributed in such a way as to assure everyone has access to housing. Such arguments deserve further consideration, irrespective of the cost-effectiveness of particular homeless assistance programs.

Opportunities for More Periodic and Systematic Use of Homeless Assistance Program Utilization and Effectiveness Data

Research and other evaluation projects are by necessity time-consuming. They are also intended to produce information that will be useful on a long-term basis for service system planning. By contrast, many of the information needs of public agencies are much more immediate. Managers and policymakers need timely data to forecast budgets, monitor their inventory of programs, guide programs toward intended policy and program objectives, and allocate resources in the most effective manner possible. Establishing accountability on the part of public agencies and the contractors with whom they work is also critical to garnering public confidence, and the willingness of legislators and the executive branches of government to continue to support these programs. Much progress has been made in the last eight years in creating means by which government agencies can track the utilization of homeless assistance programs and measure program performance. Innovations have also been achieved in the capacity and methods for measuring the impact of homelessness on other social service systems. In this section, we provide an update on advancements made since the publication of the accountability framework in 1998. We also consider some of the barriers that have been encountered in trying to implement accountability systems, and we examine two examples of jurisdictions (Arizona and Columbus, Ohio) that have been effective in using performance measures to shape their service systems to meet explicit policy objectives.

Utilization Monitoring and Reporting

When Congress created the permanent housing set-aside within HUD’s McKiney-Vento programs in 2000, it also directed the Department of Housing and Urban Development to require that grantees implement homelessness services management information systems, or HMIS. Congress asked that HUD fund implementation of such systems so that jurisdictions could establish unduplicated counts of clients served by their local homelessness services system, the characteristics of the people served, their patterns of service use and lengths of stay, and the outcomes of their services use. Congress also directed that such systems be used to identify how homelessness was associated with mainstream social welfare systems, both to assure access to mainstream services for people who are homeless and to assess whether mainstream service systems are shifting clients and responsibilities onto the homeless system. In 2000 and in subsequent appropriations, the Congress also asked HUD to compile the HMIS data into an annual report on the utilization of homeless assistance programs and their outcomes. These directives have helped the field to move toward realization of one of the primary recommendations from the Culhane et al. (1999) paper on accountability, namely, the widespread adoption of automated systems for tracking the use of homelessness services.

This effort has not been without obstacles. Most communities have still not fully implemented their systems, and many communities have struggled with technical, cultural, and human resource challenges. However, substantial progress has been made, and the HMIS initiative promises to provide much more comprehensive information for policymakers at all levels of government than has been possible before, and with it, a greater ability to plan and achieve policy goals.

As noted by Congress, one of the principle uses of these new data sources is to measure utilization and outcomes in the homeless system. At a local level, HMIS has enabled communities to create reports like the HUD Annual Progress Reports (APR) on a routine basis. It has also enabled some system managers to monitor utilization through live “dashboards” that show current trends in vacancies, length of stay, admission rates, etc. Agency planners have also been able to use data showing historic utilization patterns to forecast bed demand and as the basis for budget requests. These are but a few of the practical uses of HMIS data for performing the most elementary aspects of program administration.

HUD has also used the expanding HMIS infrastructure as a basis for creating the annual profile of homeless system utilization requested by Congress, through a project called the Annual Homeless Assessment Report (AHAR). The AHAR project has thus far involved several efforts intended to standardize information and reporting and to enable uniform assessments of homeless assistance program utilization nationally. A comprehensive set of federal data definitions and standards was published in the July 30, 2004 issue of the Federal Register. The data standards help ensure that all homeless shelters are collecting the same information on the clients they serve. Software vendors and other system developers use the federal standards to assure that their products are sufficient for meeting HUD’s expectation for privacy and data security, that common data elements are being collected and in standard formats, and to assure that key analysis and reporting features can be supported. The data elements distinguish between “universal data elements” (a short set of identifiers and characteristics to be tracked for all clients in all programs), and “program data elements” (a longer list of client characteristics and needs/progress updates that support collection of data required for HUD’s APR).

The primary goal of the AHAR project is to produce a national estimate of the number of homeless persons sheltered and a profile of the persons served. The report is based on a nationally representative sample of jurisdictions in the U.S. The 15 largest cities were all selected with certainty to be part of the national sample, as the largest jurisdictions contain a substantial proportion of the urban homeless population overall. The remaining sites were chosen randomly within strata defined by geographic region, jurisdiction type, and population. In total, 80 communities were selected for inclusion in the AHAR project.

The first AHAR report had to be based on a partial year (February 1–April 30, 2005), as HMIS implementation had not progressed sufficiently and data standards had not been promulgated in time to include the entire government fiscal year as the reporting period. A longer, but still partial period has been selected for the second AHAR (January 1–June 30); the third AHAR (September 2006–October 2007) and subsequent AHARs will cover an entire year. The 2005 AHAR includes data from 64 jurisdictions, including 55 from the sample and 9 voluntary contributors. Only jurisdictions from the sample that had sufficient participation of providers were included (50 percent of a jurisdiction’s beds had to be tracked). This “coverage” threshold was set to support reliable extrapolation for the untracked beds. 

The results of the first AHAR have not yet been published as of January 2007, but initial indications are that the results show that the HMIS effort and the national sample are effective methods for estimating homelessness nationally, and that they will eventually enable the measurement of changes in utilization over time. As more cities are able to provide data, estimation will improve. Although initial AHARs are focused primarily on population estimates and demographic breakdowns, future reports will be able to provide more detailed analyses of stay patterns and outcomes for specific subpopulations and programs.

Although the HMIS is the largest and most ambitious homelessness-specific reporting system, other federally supported reporting systems also hold promise for providing data on service utilization trends, costs, and performance measurement. These include the Department of Veterans Affairs national reporting systems, which have served as the basis for many valuable research and evaluation projects; the Runaway and Homeless Youth Management Information System (RHYMIS); and the national reporting requirements for Programs for Assistance with the Transition to Housing (PATH) projects. Each of these efforts can help to shape public awareness and understanding of homeless people, and the programs that serve them, as well as serve as tools for improving program performance and outcomes.

Administrative Data Linkages

Perhaps the most valuable use of HMIS data beyond its reporting functions is the data capacity it creates for longitudinal, multisite, multisystem research. The Congressional directive authorizing the HMIS initiative refers to the need to use HMIS data to determine if people served in homeless assistance programs are accessing mainstream social welfare and to determine if mainstream systems are shifting people and costs onto the homeless assistance system. Indeed, this may prove to be the most powerful use of HMIS data if it can encourage larger service systems to dedicate additional resources to this vulnerable population. Administrative data integration projects, or “data linkage” efforts, are not without challenges. Yet, as the surge in cost and cost offset studies mentioned previously reveals, communities are getting increasingly savvy about how to access these data sources and have had some significant successes, even without full-scale HMIS data infrastructures. In this section, we briefly consider the potential opportunities for administrative data linkages and some of the challenges that have to be overcome.

A potential research agenda for advancing our understanding of homelessness based on data integration efforts has been summarized elsewhere (Culhane & Metraux, 1997). Among the most fundamental issues to address is the degree to which the homeless system and other social welfare institutions share common populations. From the perspective of the mainstream systems, particularly those that invest heavily in institutional care (hospitals, foster care, corrections), the rate at which people leaving their care become homeless would presumably be of keen interest. From the perspective of homeless assistance systems, an important issue is the amount of shelter demand that is accounted for by people exiting mainstream systems. In both cases, researchers could use event history analysis to inform these issues and to identify risk factors that distinguish these subpopulations from their respective reference populations. Administrators could conduct periodic database merges to assess whether efforts intended to reduce discharges to homelessness are working.

A second general class of questions relates to the impact of homelessness on other service systems. The cost studies reviewed earlier are an example of these efforts. The cost offset studies are a related use, serving evaluative purposes associated with a given intervention. Evidence of a particular type of system use (e.g., inpatient mental health treatment) is also an indicator that can be used in various research projects, as a control variable, or as a moderating variable in models seeking to examine utilization dynamics or program effectiveness.

A cross-system utilization analysis could also be used to determine program eligibility — for example, for programs that target high service users. However, in most cases, uses of data integration are restricted by law to planning, auditing, and research functions and cannot be used for client contact or eligibility determination or any other means of identifying individuals, unless clients provide written consent.

The social welfare systems with data that could serve as valuable linkage include, but are not limited to: public assistance, various health service records, corrections, vital statistics, public and assisted housing, criminal justice, child welfare, public education, and earnings. Linkage with each of these data sources could form the basis of mainstream program targeting, program design, evaluation, and policy analysis across a wide variety of program areas.

Finally, address data can be used to study patterns of residential instability and moves among households that become homeless. Addresses provide a spatial distribution of the places people lived before they became homeless. This can be used as a means of studying underlying causal processes in neighborhoods or in the housing market more generally, and for geographic targeting of prevention programs. Through integration with other housing databases, researchers can also examine building- or unit-level risk factors or triggering events (e.g., utility terminations) that may present opportunities for intervention.

Research of this nature is not possible without the cooperation of the agencies that have responsibility for maintaining these data. Obtaining data access can be very complicated. However, the federal government could provide incentives or even requirements for routine data matches through its mainstream programs. For example, the federal mental health block grant program already requires states to report how many of the people with severe mental illness in their respective states are homeless and what mental health services are provided to them. It is possible that this could be answered more precisely and consistently through a database merge, perhaps on an annual basis. State Interagency Councils on Homelessness, formed in some states in response to the federally sponsored “Policy Academies,” could be the entities that use such data for their own planning and priority setting. The federal government could pilot data merge projects among willing state volunteers to demonstrate the feasibility and cost of requiring such reporting of all grantees. Similar approaches could be taken to improve state reporting regarding homeless children, prisoners re-entering society, and youth aging out of foster care.

Given their relatively low cost and temporal efficiency, administrative data linkage projects based on HMIS implementations could well be the basis for a rapid expansion of research on homelessness and on the accountability and effectiveness of homeless assistance programs. Indeed, based on the recent experiences reported here with 10-year plans, such an expansion appears to be already underway. However, as has been observed in the growing number of cost and cost offset studies, many of these efforts could benefit from the participation of academic partners and from federal support. Organized and sponsored programs of research are necessary to bring needed cohesion and value to these and other projects like them.

Until now, we have focused on the literature and reporting tools that inform system design, policy, and program planning. Another area in which there has been some progress since 1998 is program assessment and performance measurement. While most communities are still working to implement their HMIS, some communities have gone further by using HMIS and other program data to assess how programs are doing relative to one another in terms of client outcomes. A few others have used such data to award performance incentives to programs that meet stated objectives, such as improved housing placement rates, or shortened lengths of stay (“performance-based contracting”). Such uses of HMIS and program performance data provide homeless assistance system administrators with systematic tools with which they can attempt to manage or shape provider behavior. Such tools can help to assure that programs are working to serve designated client populations, delivering the intended services and achieving the desired outcomes. While some promising practices have emerged in this area, fully operational models are still far and few between. Only a few of the larger and more sophisticated homelessness service systems are likely to include ongoing performance assessments, let alone performance-based contracting.

In a recent overview of outcome measurement in homeless assistance programs, Crook et al. (2005) characterize an outcome measurement system as “a comprehensive, systematic approach to identifying, tracking, and reporting data that reflect the extent to which program participants experience the intended benefits or changes as a result of service provision” (p. 379). However, the authors state that they were unable to locate a single comprehensive outcome measurement instrument that could be used for the homeless assistance system of care. Instead, at the client level, there are instruments that reflect the impact on a single domain, primarily mental illness or substance abuse. In this section, we review the efforts of a model program from Arizona, where an assessment and outcome system was created that is giving providers the ability to better measure whom they serve and how they perform in terms of client progress over time. A feedback system helps providers to benchmark their effectiveness relative to other providers, and to meet and discuss program strengths and weaknesses. Following that case study, we will also examine the experience of Columbus, Ohio, where regularly collected and analyzed program data has enabled that city to shape its service system to meet stated policy objectives.

Case Study: The Arizona Evaluation Project on Homelessness

The Arizona Evaluation Project on Homelessness was designed to address the need to improve the measurement of program impacts at the client level. The Project was designed to use aggregate impact measures to assess the effectiveness of particular agencies as well as the overall effectiveness of the various continuums of care in the state. The Project commenced in 2002 and included several stages, including an assessment of best practices in outcome measurement, psychometric testing of various instruments, the creation and deployment of a standardized instrument, establishment of a reporting and analysis system, and the creation of a feedback process with the providers.  

The first stage brought together service providers to determine what, if any, evaluation tools were being employed by their agencies. Each agency that provided an instrument was also asked to provide raw data on at least 150 homeless clients. The intent was to analyze the psychometric properties of the existing instruments to determine which, if any, met sufficient standards for reliability and validity. Approximately ten instruments were provided, half of which were called “Self-Sufficiency Matrix.” Despite the common name and some obvious similarities across the instruments, the various self-sufficiency matrices had striking differences and appeared to reflect different evolutions at each agency of a long-lost progenitor tool.

Reliability is measured in many ways and is often narrowly defined as the extent to which two measurements yield consistent results in a short period of time (test-retest reliability). This is a specific type of reliability, but the concept of reliability is broader; it also refers to the amount of error in a given set of measurements. The type of reliability most often studied by psychometricians is internal reliability, which measures the level of error and hence the quality of a given instrument. The internal reliability of each assessment tool provided to the project team was assessed using the archived data set accompanying the tool.

While an instrument can be reliable, it may still not be useful. To help assess the potential utility of each of the assessment tools provided, construct validity was also examined. To examine the extent to which the instruments were capturing one or more underlying constructs, a factor analysis was also conducted for each instrument. A factor analysis is a multivariate statistical technique which determines the extent to which items on a test “clump” together to form subsets of questions that measure particular scales. Identifying the existence of such underlying scales can be used to establish client typologies for program targeting as well as program performance assessment.

Upon review of the ten instruments that were submitted along with archived data, only one instrument met acceptable reliability and validity standards. This tool was one of the versions of the “self-sufficiency matrix”; it was far superior not only to the other types of instruments but also to the other versions of the self-sufficiency matrix. Since this instrument showed some promise, it was further piloted by a number of local agencies for six months. The agencies submitted all of their data for further psychometric testing. One large agency used the tool as a client self-report measure, while the others used it as a case manager reporting tool. Results from the pilot indicated that it was an inappropriate tool to use for self-report with the homeless population, but it was much more reliable and valid as a case manager reporting tool. The factor analysis yielded two robust factors: the extent of client dysfunction/functioning and the extent of independent life skills. An overall combined score for self-sufficiency is the sum of these two factors. The two factors and the overall score all demonstrated good reliability (internal reliability of client dysfunction=.79, independent life skills=.78, and overall self-sufficiency = .81). The final instrument produced is provided in Exhibit 1.

The client assessment tool was then used for predictive mathematical modeling. The fear of the project staff was that building expectations and incentives for demonstrating client improvement alone could produce an unintended consequence, namely, that agencies would gravitate toward the “low hanging fruit,” i.e., relatively easy clients who require less investment of staff time to produce results. An assessment system that included disincentives to serve a particular client group would be counterproductive. The predictive modeling was an attempt to avoid this dilemma. Using HMIS data fields including supplementary client history fields and baseline scores on the self-sufficiency matrix, equations are generated to determine the predictors of change while in homeless assistance programs for the varying level of dysfunction, independent skills, and overall self-sufficiency. These equations are then used to predict the amount of change that would be predicted in each individual client if randomly assigned to a homeless assistance program. Each individual’s predicted change is uniquely determined based upon the client’s individual characteristics. These predicted changes constitute the expected change for each client. The expected change is then compared to the actual change at the time of program exit. Agencies whose clients typically do better than expected are the most successful and those whose clients typically perform below expectations are in need of programmatic improvements (see sample feedback form, Exhibit 2).

Exhibit 2 Arizona Homeless Evaluation Project Progress Report
(Based on clients who have exited the program; N=129)

I.    PROGRAM:                Demo Shelter
      Type:                       Emergency Shelter
      Continuum:             MAG Continuum of Care Regional Committee on Homelessness
      Date:                       June 19, 2006

We have compared characteristics of Demo Shelter clients to clients from other agencies with like program types within the Maricopa Continuum of Care who entered and exited programs during the same time period (October 2005 through March 2006). In terms of these demographic variables, Demo clients tend to be mildly older, mildly less likely to be female, and mildly more likely to serve black clients and mildly less likely to serve Hispanic clients. However, overall there are not great differences in the demographic characteristics.

The predictive model determines the most likely change each client would make if they were randomly assigned to a homeless assistance program. This expected change is then compared to the actual change clients make in the program. If the difference is positive this program is performing above expectations and if the difference is negative then the agency is performing below expectations.

Overall, Demo Shelter is mildly better than other programs in decreasing dysfunction and moderately better in increasing independent life skills and overall self-sufficiency. Demo Shelter has its greatest success with homeless individuals recently released from jail/prison. An area of challenge for Demo Shelter is the program’s difficulty in having significant impact with its Hispanic clients.

No agency excels with all clients, and the predictive model allows each agency to objectively explore whether there are systematic differences between the types of clients with whom they experience the most success and those who are most challenging. Each agency receives a written feedback report on a quarterly basis detailing how, if at all, their clients differ from those served by other agencies, the extent to which agency outcomes differ from those expected from the predictive model, and the relative strengths and weaknesses of client successes within each agency. For example, one agency serving disabled and older homeless men and women was able to determine that it was far more effective with the older subpopulation than with people with disabilities. Further analyses showed that the frequency of “acting out” behavior among the people with disabilities was determinative of agency effectiveness, with a greater frequency of “acting out” associated with less successful client progress. This agency is now exploring what practices and techniques can increase its effectiveness with such clients. Another agency was able to identify that despite stronger outcomes than expected overall, it was much less successful with Hispanic clients. As a result, the agency is working with agencies that are more successful with Hispanics to help identify what changes might increase its effectiveness with this subpopulation.

Such feedback systems can also allow agencies to rethink their target populations. If an agency learns that it is effective with people who have a mental illness or a substance use disorder, but is ineffective when these conditions are co-occurring, that knowledge is valuable both for the program and for the local continuum of care. For example, if another agency is highly effective with clients who have co-occurring disorders, the initial agency can either choose to learn from that agency and strengthen outcomes with this group, or it can decide to accept clients with whom it is likely to be effective and refer those clients with whom it is less likely to be successful to programs more likely to benefit them.

The initial expectation of the project was that agencies would naturally discuss and learn from each other in this feedback process. However, it became apparent that the various continuums of care (CoCs) could play a convening role by structuring activities that brought both leadership and line staff from the agencies together to learn from each other in “evaluative learning circles.” These are regularly scheduled meetings of homeless agencies from similar locations with similar missions to learn from each other the relative strengths and weaknesses of each and how they can cooperate to produce better client outcomes.

Beyond aiding individual clients or individual agencies, the evaluation system has been helpful in identifying patterns that are valuable for policy considerations for the CoCs as a whole. One finding has been that the distinction between emergency and transitional programs in actual practice in Arizona appears to be an arbitrary one. There is no difference locally between the two types of programs in who they serve, the types and extent of problems their clients exhibit, or the expected change from each program. Another finding in data analyzed thus far suggests that, across all agencies, there is a window of between three and seven weeks when programs are likely to have their greatest impact. Shorter term stays are typically inadequate to effect change, and stays longer than seven weeks tend to cause individuals (but not families) to regress. This suggests that, for homeless individuals, a period of training and stabilization of three to seven weeks followed by placement in long-term housing is likely to maximize client impact. It is also hoped that the predictive model will assist in the rating and ranking process for the McKinney-Vento Assistance application by making quality assessments more objective and rigorous.

The findings related to duration of treatment and lack of distinction between emergency and transitional programming were included to demonstrate the types of findings the model is capable of yielding. However, these results should be regarded with some caution. They are accurate for the sample of homeless we have studied. The sample is not yet representative of the broader homeless community and a sizable number of clients in transitional housing are still in the pipeline without yet having an exit matrix. We are anxious to see if these findings persist when the dataset becomes more representative of the entire state homeless population.

This case study provides one example of how a jurisdiction is able to use program and outcomes data to develop benchmarking and performance standards, as well as to develop a process for engaging providers in discussions about strategies for improving their performance. The development of the self-sufficiency matrix was an important tool in that process, as was the creation of learning communities. Other potential approaches are also possible. In the next section, a case study from Columbus, Ohio, is presented, with particular attention to some of the challenges that community faced in bringing performance measurement to its system.

Case Study: The Community Columbus Shelter Board, Columbus, Ohio

Since 1997 the Community Shelter Board has conducted annual program evaluations for the Columbus and Franklin County Continuum of Care Steering Committee. The Steering Committee utilized evaluation of renewing projects as a means to make ranking decisions, adjust funding awards, and monitor program performance. The program evaluation has considered client characteristics, program utilization and outcomes, program design and implementation, and program costs. The evaluation compared planned results as described in the prior application with actual results obtained. The program was also assessed for compatibility with local priorities and overall community impact. The data were obtained from HUD APRs, interviews with providers, and on-site program visits. Over time, the Steering Committee began tracking and comparing housing outcomes for all programs, as well as comparing program costs (per household served and per housing unit provided). As the process is centered only on the HUD application and is not part of the HUD contracting process, it has more control than a purely voluntary process but less control than a performance-based contracting process. (See Exhibit 3.)

Over this 10-year period, the evaluation process has been modified to better address community needs, respond to best practices, and comport with HUD funding requirements. The impact of using data to inform community funding decisions has been profound:

1. Overall program performance has increased. Programs experience higher housing outcomes and improved program occupancy, and serve more challenging clientele.
2. The inventory of programs has shifted to 91 percent permanent supportive housing beds in 2006 vs. 69 percent in 1997.
3. Community confidence in program accountability and results has increased.

As a result of poor program performance, the Steering Committee ended funding for eight transitional housing and supportive services only programs. Additionally, three programs converted from transitional housing to permanent supportive housing. The latter occurred as the Steering Committee determined that HUD continuum-of-care resources could be allocated on a priority basis to programs that focus both on (1) high need clients (i.e., those with long histories of homelessness, severe disabling conditions, and limited income); and 2) improved housing outcomes for those clients. Clients with low needs (i.e., those with fewer barriers to housing placement, less disabling conditions, and/or better income stability) were diverted to housing placement services and community-based services that were both more effective at meeting their needs and less expensive to the community.

The Steering Committee established a priority for effective and innovative housing service delivery that is expressed as providing housing and services for those with the greatest needs and greatest difficulty accessing the current homelessness service system. Monitoring of program admission and client selection practices has been particularly important during evaluation to determine how programs serve persons with special needs, demonstrate proactive inclusion and non-restrictive housing admission requirements, and practice expedited admission processes. Thus programs that operate in a more selective manner, such as requiring multiple interviews, mandating pre-admission drug testing, and/or restricting admission by persons with criminal histories will disadvantage those with histories of chronic homelessness and multiple barriers. Such program would be rated lower in performance. Based on these provider ratings, HUD resources can be prioritized for the most difficult to house homeless persons.

The Steering Committee has defined program occupancy as one measure of cost-effectiveness. The average monthly occupancy over the 12-month review period should be at least 95 percent. Low occupancy can indicate many program problems, including offering a program that is not desired or needed by homeless persons, selective admission practices, and/or poor property management resulting in slow unit turnover. By evaluating occupancy, the Steering Committee pushed providers to adjust their practices to assure that the precious resource of housing was available to homeless persons on a timely basis.

As HUD has only recently defined housing stability measures (as opposed to allowing programs to self-define outcomes), it was necessary for the local Steering Committee to define the measurement and assign a performance target.  The Steering Committee established that as all HUD funding programs were aimed at addressing the needs of homeless persons, it was imperative that housing stability be a primary outcome for each program. This shift is evident when comparing residential stability goals in the late 1990s to the most recent period.

For example, a Shelter Plus Care provider was operating under these agency-designed residential stability goals during 1998-99:

1. 50 percent of initial participants will maintain continuous sobriety and active participation in all program components for at least their first 12 months.
2. 50 percent of the single women clients who had children placed in foster care prior to entry into the program will regain custody within 12 months of program entry.
3. 100 percent of clients will develop quarterly goals for independent living skills.

In 2006, this same program was required by the Continuum of Care Steering Committee to meet the following residential stability goals:

1. There is evidence in the APR that at least 80 percent of persons served during the evaluation period remain in the permanent supportive housing project or exit and move into permanent housing, where the client has control of the housing.
2. The average length of stay for persons living in permanent supportive housing is at least 12 months.
3. The project has met its housing stability goals for the APR period being evaluated.

This example illustrates the shift from addressing homelessness as a personal condition in need of rehabilitation to addressing homelessness as a condition resolved by achieving housing stability. In 1998-99, this program would have considered clients to have been “successful” if they were sober but still homeless. In 2006, clients are only “successful” if they remained housed and are no longer homeless.

The full evaluation report includes all programs that were evaluated during the period and is provided to each agency for distribution to program and management staff. It is hoped that agency leadership not only shares the report but also uses the measures to communicate their vision for program and client outcomes. The ability to benchmark programs against other programs operating within the community is also helpful.

Program and financial data were readily available to the CoC Steering Committee due to the HUD requirements for submission of annual reports. Upon closer review, we did find that programs that experienced program and agency administrative problems were not able to produce reliable, accurate client and financial data. The lack of administrative capacity was also usually correlated with poor program performance.

Providers have resisted the use of standardized measures, citing concerns about differences in admission criteria, program design, and resources. Initially, some providers were more focused on service and treatment delivery, rather than housing stability, thus they were resistant to having their programs’ performance evaluated on the basis of attainment of stable housing.

The conduct of annual program evaluations is also not without cost. The Steering Committee’s process requires the services of an outside evaluator and two or three Steering Committee members who participate in the site visits. The evaluator is responsible for reviewing program documentation and reports, communicating with the provider, coordinating and participating in site visits, and summarizing findings. Providers also absorb staff costs related to preparing for the evaluation, participating in site visits, and responding to the reviewers’ report.

As renewal grants are now required to be limited to one-year terms, rather than three- to five-year terms, the number of programs reviewed is increasing each year. The need for annual program evaluation is being questioned, as overall program performance has improved over time and nearly all programs consistently perform at high levels. The Steering Committee is considering the efficacy of conducting bi-annual reviews for high performing programs and reserving annual program evaluations for programs with sub-par performance.

Another challenge relates to the timing of the design of program evaluations. All too often programs are designed for implementation, with evaluation measures as an afterthought or treated only as a grantor-imposed requirement. Thus, program evaluation measures may be perceived as irrelevant to the program, not measurable based on data collection instruments, and/or too costly for implementation.

Another challenge is that programs change over time and their evaluation methods may not change. The Steering Committee observed the latter when a program shifted from an abstinence-based sobriety housing model to low-demand safe haven programming. Obviously, attainment and maintenance of sobriety was no longer relevant as a measure of self-sufficiency, but measuring reductions in substance use, while more relevant, was also more difficult. This particular provider was also reluctant to concurrently reduce admission barriers (be less selective in admission) and increase housing outcomes expectations as it believed that serving a more “difficult” population would mean that housing outcomes would decrease. Based on local experience and the national literature, however, the Steering Committee required that housing outcomes goals be greater than under the prior program design.

Recently, the Community Shelter Board has begun publication of quarterly program indicator reports from the HMIS. Most HUD SHP–supported programs submit data into the HMIS, and Shelter Plus Care programs will be added over the next year. The following measures are reported for each program:

1. Number served
2. Program occupancy (average number of units occupied)
3. Housing stability (average length of stay)
4. Housing outcomes (number remaining in supportive housing or moved to other permanent housing destination)

Results are compared to community or program standards (if higher than community) for compliance. CSB also aggregates data across programs to create a report on results for the systems as a whole (i.e., family shelter, adult shelter, and supportive housing). In the future, CSB intends to include clients’ demographic and key characteristics (gender, age, race, household type, disability, education, homelessness history, etc.) to better understand program results. As the shelter and housing systems better refine their assessment processes, it will be possible to better define risk adjusted outcome targets and improve matching between programs and clients.

To provide accountability to the community and promote transparency, CSB posts all program evaluations and indicator reports to www.csb.org . This transparency has been very powerful in achieving greater program and system accountability for client results. While some providers have expressed concern about this practice, it is overwhelmingly supported by funders, providers, and others. Although there was concern about the potential for political fallout (e.g., loss of local government funding) if programs did not achieve planned results, this has not been the case. Continuously low-performing programs have improved program performance, changed the program model, or ended the program. The elimination of programs has been both voluntary and as result of funding withdrawal. The overall result is better-performing programs that address higher priority community needs.

By focusing on a limited number of indicators that are directly related to the overall community goal of ending homelessness, it is feasible to utilize the HMIS to report on the impact across programs and for the overall system of care. This approach could be feasible for communities across the country to implement. While providers may want to track and report on other measures, e.g., completion of treatment, job placements, etc., these measures would vary by program and thus be difficult to implement across all programs. By keeping the approach simple, communities will be more successful at implementation and will be more effective at communicating progress and challenges to the public and decision makers.

As the Columbus experience illustrates, creating accountability systems and performance measures is possible, but not without challenges. Including providers, funders, and other community leaders in the process can help to encourage change, and transparency can assure that problems and issues are confronted in an open and forthright manner. Most importantly, the Columbus experience shows how deliberate goal setting accompanied by consistent and clear performance measurement can be used to move both providers and the service system overall in a desired policy direction and, ultimately, change the configuration of the service system consistent with the goals of the local planning authority. Since most agencies have multiple funding streams, it is important that performance measures be constructed to allow the agencies to respond to a variety of grant reporting requirements. It is important to construct the measurement system so that the basic measures (stable housing, employment and/or income, linkage to needed services like mental health, and improvements in education/skills) be used to respond to multiple grants.

Summary and Conclusion

Since 1998, progress has made in our understanding of how homeless assistance programs could be more cost-effective and more responsive to consumer needs; however, much more remains to be done. Considerable research has been conducted that shows that various supportive housing models are effective for ending homelessness among most people with severe mental illness. For people with histories of heavy service utilization, these interventions are likely to achieve significant offsetting cost reductions, or at least cost neutrality. This literature also suggests that service matching and other program targeting strategies are also indicated. Most homeless people with mental illness, even those who are not the target of experimental interventions, have short-term homelessness and positive housing outcomes, suggesting that a smaller subsegment of this population needs the intensive (and more expensive) housing and service interventions that have been tested in the literature. Research on chronic homelessness likewise suggests that a small subsegment of the homeless population consumes most of the homeless system resources and is likely to be unable to exit without significant housing and service supports. Thus, while not all people who experience chronic homelessness have severe mental disorders nor are they all heavy service users (service use may vary as a function of regional and other accessibility factors), it is likely that many are costly users of public services, including homeless system resources, and therefore, they would be the appropriate targets of the more intensive supportive housing interventions.

People who experience non-chronic homelessness, including most families and the vast majority of homeless people overall, would seemingly require less intensive interventions. Unfortunately, this is an area where the literature is quite limited. Research is needed to identify the various costs associated with these subpopulations, in part to inform the potential cost efficiency of alternative program models. Relocation programs, transitional rental assistance, and various service support models may be effective in reducing or preventing homelessness among these subpopulations, and future research could test such models. These can include programs specifically targeting people transitioning out of institutions, people with substance use disorders, and people with temporary economic or domestic crises. While cost-effectiveness or cost offsets may or may not be achieved, such research would identify if better outcomes can be achieved than from congregate shelters, and more efficiently.

Significant progress has been made in the area of standardized, automated information collection on homeless assistance program use. HUD’s HMIS initiative has led to the adoption of client tracking technology in hundreds of jurisdictions, and with sufficient coverage for jurisdiction-wide reporting in several dozen cities to date. Future research could take advantage of these data for local studies of homelessness service utilization patterns, as well as for analyses of multi-system services use and costs. More than 30 studies have recently attempted to track costs and cost offsets associated with chronic homelessness through the analysis of multiple service system databases. These efforts could be further expanded and standardized with appropriate federal support, and should take advantage of the implementation of HMIS programs in communities around the country.

The expansion of HMIS capacity has also made possible more rigorous program performance assessments. In this paper, best-practice case studies from Arizona and Columbus, Ohio, were provided that illustrated how these communities were able to implement a client assessment and tracking system that also formed the basis for measuring provider performance. Providers can be measured with regard to a peer group, and their outcomes tracked and compared over time. In Arizona, a process has been established whereby the agencies can share successes and strategies for program improvement, based on their quarterly performance reports. And in Columbus, yearly reviews by the Continuum of Care Steering Committee set expectations and goals for providers, and monitor annual progress in meeting those goals. Such systems hold the promise of making programs more accountable to consumers by assuring that target populations are served (not underserved), that the intended services are delivered, and that they are having their expected outcome. In so doing, a feedback process can be created that will help providers to continually improve their programs. Creating accountability systems is not without challenges. Some providers will be resistant to program performance measurement and to changes that may be required based on feedback. But including relevant stakeholders and an open process can help to insure that provider interests are addressed, at the same time that the community’s priorities can be achieved.

References

Caton, C. M., Dominguez, B., Schanzer, B., Hasin, D. S., Shrout, P. E., Felix A., et al. (2005). Risk factors for long-term homelessness: Findings from a longitudinal study of first-time homeless single adults. American Journal of Public Health, 95, 1753–1759.

Clark, C., & Rich, A. (2003).Outcomes of homeless adults with mental illness in a housing program and in case management only. Psychiatric Services, 54, 78–83.

Crook, W. P., Mullis, R. L., Cornille, T. A., & Mullis, A. K. (2005). Outcome measurement in homeless systems of care. Evaluation and Program Planning, 28, 379–390.

Culhane, D., Eldridge, D., Rosenheck, R., & Wilkins, C. (1999). Making homelessness programs accountable to consumers, funders and the public. In L. B. Fosberg & D. L. Dennis (Eds.), Practical lessons: The 1998 National Symposium on Homelessness Research. Washington, DC: U.S. Department of Housing and Urban Development and U.S. Department of Health and Human Services.

Culhane, D. P., & Metraux, S. (1997).Where to from here? A policy research agenda based on the analysis of administrative data, In D. P. Culhane & S. P. Hornburg (Eds.), Understanding homelessness: New policy and research perspectives (pp. 341–360). Washington, DC: Fannie Mae Foundation.

Culhane, D. P., Metraux, S., & Hadley, T. (2002). Public service reductions associated with placement of homeless persons with severe mental illness in supportive housing. Housing Policy Debate, 13, 107–163.

Culhane, D. P., Metraux, S., Park, J. M., Schretzman, M., & Ventura, J. (in press). Testing a typology of family homelessness based on patterns of public shelter utilization in four US jurisdictions: Implications for policy and program planning. Housing Policy Debate.

Dickey, B. (2000). Review of programs for persons who are homeless and mentally ill. Harvard Review of Psychiatry, 8, 242–250.

Dickey, B., Latimer, E., Powers, K., Gonzalez, O., & Goldfinger, S. (1997). Housing costs for adults who are mentally ill and formerly homeless. The Journal of Mental Health Administration, 24, 291–305.

Dunford, J., Castillo, E., Chan, T., Vilke, G., Jenson, P., & Lindsay, S. (2006). Impact of the San Diego Serial Inebriate Program on use of emergency medical resources. Annals of Emergency Medicine, 47, 328–336.

Gladwell, M. (2006, February 13). Million dollar Murray. The New Yorker.

Homeless Management Information Systems (HMIS); Data and Technical Standards. Final Notice. 69 Fed. Reg. 45888 (July 30, 2004).

Jones, K., Colson, P. W., Holter, M. C., Lin, S., Valencia, E., Susser, E., & Wyatt R. J. (2003). Cost-effectiveness of critical time intervention to reduce homelessness among persons with mental illness. Psychiatric Services, 54, 884–890.

Kuhn, R., & Culhane, D. P. (1998). Applying cluster analysis to test a typology of homelessness by pattern of shelter utilization: Results from the analysis of administrative data. American Journal of Community Psychology, 26, 207–232.

Kuno, E., Rothbard, A. B., Averyt, J., & Culhane, D. (2000). Homelessness among persons with serious mental illness in an enhanced community-based mental health system. Psychiatric Services, 51, 1012–1016.

Kushel, M. B., Perry, S., Bangsberg, D., Clark, R., & Moss, A. R. (2002). Emergency department use among the homeless and marginally housed: Results from a community-based study. American Journal of Public Health, 92, 778–784.

Lehman, A. F., & Cordray, D. S. (1993). Prevalence of alcohol, drug, and mental disorders among the homeless: One more time. Contemporary Drug Problems, 20, 355–83.

Lehman, A. F., Dixon, L., Hoch, J. S., DeForge, B., Kernan, E., & Frank, R. (1999). Cost-effectiveness of assertive community treatment for homeless persons with severe mental illness. British Journal of Psychiatry, 174, 346–352.

Lennon, M. C., McAllister, W., Kuang, L., & Herman, D. (2005). Capturing intervention effects over time: Reanalysis of a critical time intervention for homeless mentally ill men. American Journal of Public Health, 95, 1760–1766.

Lindblom, E. (1991). Towards a comprehensive homeless-prevention strategy. Housing Policy Debate. 2, 957–1025.

Lutterman, K., & Hogan, M. (2000).State mental health agency controlled expenditures and revenues for mental health services, FY 1987 to FY 1997. In R. Manderscheid & M. Henderson (Eds.), Mental heath, United States, 2000. Rockville: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services.

Martinez, T. E., & Burt, M. R. (2006). Impact of permanent supportive housing on the use of acute care health services by homeless adults. Psychiatric Services, 57, 992–999.

Maguire, M., Culhane, D. P., Metraux, S., & Poulin, S. (2005, September 14). Using HMIS to measure chronic homelessness, service use patterns and costs. Paper presented at National Homeless Management Information System (HMIS) Conference, St. Louis, MO.

National Alliance to End Homelessness. (2006). Promising strategies to end family homelessness. Washington, DC: Freddie Mac.

Rosenheck, R. (2000).Cost-effectiveness of services for mentally ill homeless people: The application of research to policy and practice. American Journal of Psychiatry, 157, 1563–1570.

Rosenheck, R., Kasprow, W., Frisman, L., & Liu-Mares, W. (2003). Cost-effectiveness of supported housing for homeless persons with mental illness. Archives of General Psychiatry, 60, 940–951.

Rothbard, A. B., Min, S., Kuno, E., & Wong, Y. I. (2004). Long-term effectiveness of the ACCESS program in linking community mental health services to homeless persons with serious mental illness. Journal of Behavioral Health Services and Research, 31, 441–449.

Salit, S. A., Kuhn, E. M., Hartz, A. J., Vu, J. M., & Mosso, A. L. (1998). Hospitalization costs associated with homelessness in New York City. The New England Journal of Medicine, 338, 1734-1740.

Shern, D., Tsemberis, S., Anthony, W., Lovell, A. M., & Richmond, L. (2000). Serving street-dwelling individuals with psychiatric disabilities: Outcomes of a psychiatric rehabilitation clinical trial. American Journal of Public Health, 90, 1873–1878.

Shinn, M., Baumohl, J. & Hopper, K. (2001). The prevention of homelessness revisited. Analyses of Social Issues and Public Policy, 1, 95–127.

Schumacher, J. E., Mennemeyer, S. T., Milby, J. B., Wallace, D., & Nolan, K. (2002). Costs and effectiveness of substance abuse treatments for homeless persons. The Journal of Mental Health Policy and Economics, 5(1), 33–42.

Sullivan, J. G., Koegel, P., Jackson, C. A., Jinnett, K., Morton, S. C., Miu, A., & Chien, S. (n.d.). Use of public sector services and associated costs among homeless adults with and without serious mental illness. Manuscript in preparation.

Susser, E., Valencia, E., Conover, S., Felix, A., Tsai, W. Y., & Wyatt, R. J. (1997). Preventing recurrent homelessness among mentally ill men: A "critical time" intervention after discharge from a shelter. American Journal of Public Health, 87, 256–262.

van Leeuwen, J. (2004). Reaching the hard to reach: Innovative housing for homeless youth through strategic partnerships. Child Welfare, 83, 453–468.

Wolff, N., Helminiak, T. W., Morse, G. A., Calsyn, R. J., Klinkenberg, W. D., & Trusty, M. L. (1997). Cost-effectiveness evaluation of three approaches to case management for homeless mentally ill clients. American Journal of Psychiatry, 154, 341–348.

Wong, Y-L. I., Koppel, M., Culhane, D. P., Metraux, S., Eldridge, D. E., Hillier, A., & Lee, H. R. (1999). Help in time: An evaluation of the Philadelphia City’s community-based homelessness prevention program. Philadelphia: University of Pennsylvania School of Social Work.

Young, A. S., Chinman, M. J., Cradock-O’Leary, J. A., Sullivan, G., Murata, D., Mintz, J., & Koegel, P. (2005). Characteristics of individuals with severe mental illness who use emergency services. Community Mental Health Journal, 41, 159–68.

Appendix: The U.S. Interagency Council on Homelessness

Studies

• Summary of Cost Offset Studies
• Intervention Studies

Endnotes

¹ While outreach as an intervention as such represents a cost of resolving homelessness, in some cases one might consider outreach a cost of homelessness, against which one could measure the associated decline in outreach costs for persons who are subsequently housed or placed in a program. Of course, because outreach is part of the interventions studied, it is appropriately considered as a cost of the interventions in comparison to doing no outreach. Where to show the costs of outreach will depend on the boundaries of the intervention studied.

² We use the term "long-term" shelters to more directly refer to their functional role in the shelter system.  Distinctions between transitional housing and emergency shelters are not always clear or obvious in practice, and sometimes reflect funding sources more than programmatic differences.  Alternatively, long shelter stays can occur in both "emergency" and "transitional" facilities, although some facilities are disproportionately used for long stays than others (Culhane, Metraux, Park, Schretzman, & Valente, in press).