Toward Understanding Homelessness: The 2007 National Symposium on Homelessness Research. Accountability, Cost-Effectiveness, and Program Performance: Progress Since 1998.. Background


In their 1998 paper, Making Homelessness Programs Accountable to Consumers, Funders and the Public (Culhane et al., 1999), the authors provide a framework for assessing program outcomes that addresses the information needs of the various constituencies for homelessness services (consumers, funders, and the public). Consumers, it was argued, need data on services received and whether those services meet their perceived needs. Funders require program-level performance data both to demonstrate that programs are delivering the services they are funded to perform, and to compare providers on standardized performance benchmarks. The public needs system-wide performance measures that demonstrate whether the system as a whole is meeting its primary objectives of improving the lives of homeless people and reducing homelessness, and to demonstrate if it is doing so in as efficient and cost-effective a manner as practicable. A variety of studies were cited that illustrated different approaches to these areas and that demonstrated that the measurement of outcomes of interest to these respective audiences was indeed possible. Nevertheless, the empirical evidence on program accountability was quite limited at that time. Little systematic research was occurring on a national scale, aside from a few federally sponsored research demonstrations and the research based on the nationally standardized program data from the U.S. Department of Veterans Affairs. The research was also limited in its timeliness and utility, in that it was frequently geared toward academic audiences, appearing in professional journals years after the completion of the interventions studied. The authors recommended an expansion of standardized data collection tools, including more brief questionnaires that would have program-level and local service system utility. They also argued for a broader, more “systems-wide” view of homeless assistance programs that would examine the relationship of homeless assistance programs to other, mainstream social welfare systems. The authors encouraged the development of automated systems that could track clients longitudinally and that could be used to assess program effectiveness on a more routine basis. To accomplish this, the authors also urged closer collaboration between researchers and practitioners.

Since the publication of that paper, important advances have been made, although much remains to be done. In general, our sense is that the rate of published research on homelessness has slowed since 1998. For our purposes here, we have chosen to focus on the areas where there does seem to be a growing interest and/or literature: analyses of the costs of homelessness and the cost offsets associated with various interventions; and the use of standardized data collection tools and performance reporting in homeless assistance programs. Unfortunately, the literature has grown unevenly with respect to the various subpopulations who experience homelessness. As has been the case for the last 25 years, since 1998 research on homelessness among people who have a severe mental illness (SMI) is more prevalent than research on other populations, perhaps owing to the funding priorities of federal research sponsors such as the National Institute of Mental Health (NIMH) or the Substance Abuse and Mental Health Services Administration (SAMHSA). The bulk of our literature review will, therefore, focus on services and housing for persons with severe mental illness, and the review is far less comprehensive for other persons who are homeless. Although there has been a recent increase in studies of chronic homelessness, beyond people with severe mental illness, this is a relatively new phenomenon; most prior research has not included people who do not have a severe mental illness, even among the chronically homeless — for example, people whose primarily disability is related to substance use — although people without severe mental illness are the majority of people who experience chronic homelessness. Hence, as a result of both the reduced rate of published research since 1998 and its continued focus on persons with mental illness, the available literature is frustratingly narrow in applicability to the homelessness problem overall.

That said, some research on families and youth has begun to inform interventions for these populations, as has literature on adults who are not chronically homeless but who are transitioning from institutional programs (prison, foster care, detoxification). Experimental tests of these interventions are not yet available, and neither are there many detailed cost and utilization studies associated with these subpopulations. Where limited data are available — for example, for families — they will be discussed here.

Despite the limited progress to be found in the research literature, policymakers have made substantial progress in increasing the accountability of homeless assistance programs since 1998. Most significantly, the requirement by Congress in 2000 that communities implement Homelessness Services Management Information Systems (HMIS) has been responsible for a broad and systematic expansion in data collection on both clients and programs. To date, actual research based on these data has been limited, as most communities have had to overcome various technical and human resource challenges during system implementation. However, a growing body of cities have successfully implemented HMIS, and as that pool grows, analyses based on these data will surely find their way into the published literature. The increasing interest in costs and the cost offsets associated with various interventions and subpopulations could also drive more research based on HMIS data. Absent information identifying emergency shelter users from HMIS, it is difficult for communities to track services use across other service systems, let alone within shelter and housing programs. One could argue that the literature on costs and utilization has not grown as quickly as one might have expected since 1998 because of the slow pace with which these information systems have been fully implemented. However, as implementations proceed, more data should yield more analysis and knowledge development, including for populations of homeless people who are not experiencing chronic homelessness.

Until the relatively recent adoption of HMIS, the lack of standardized data collection also restricted the growth of performance measurement and performance-based contracting in the field of homelessness. Performance measurement generally has been limited to fulfilling reporting requirements, such as HUD’s Annual Performance Report (APR). While such performance reports enable HUD to conform to the Government Performance and Results Act of 1993 (GPRA), they have not generally served as practical tools for program improvement and management at a local level (with some exceptions, see the Columbus case study below). A few communities have set up other reporting systems that serve as the basis for performance-driven planning and contracting. Such reporting and feedback processes are necessary for guiding service systems toward a desired set of policy objectives. The HMIS infrastructure could be used as the basis for such systems in the near future. Innovators in this area, such as Arizona and Columbus, Ohio, have shown how quality program-level information can be used for such purposes.

Culhane et al. (1999), in their “accountability” paper, established a framework for considering comprehensively how to measure the accountability of homeless assistance programs to funders, consumers, and the public. However, progress since then has been mixed, and only a subset of programs (mostly a particular type and intensity of supportive housing) and a subset of the homeless population (people with severe mental illness, and some people who are chronically homeless) have been studied to any significant degree since the last symposium. We still know very little about the accountability of other service interventions and the costs associated with homelessness among several important population groups. What follows here is an assessment of what we do know, how the field is working systematically to improve our knowledge base, and some examples of how communities can integrate accountability and management standards into their practices so that further advances can be made across a broader spectrum of programs and populations in the near future.

View full report


"report.pdf" (pdf, 561.34Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®