Core Performance Indicators for Homeless-Serving Programs Administered by the U.S. Department of Health and Human Services

Prepared for:
Office of the Assistant Secretary for Planning and Evaluation
U.S. Department of Health and Human Services

Project Officer:
Walter Leginski

Prepared by:
John Trutko
Capital Research Corporation

Burt Barnow
Johns Hopkins University

 

"

Introduction

This chapter synthesizes the results of interviews with administrators and a review of relevant program documentation at the four DHHS homeless programs that are the focus of this study: (1) Programs for Runaway and Homeless Youth (RHY) Program, (2) the Health Care for the Homeless (HCH) Program, (3) Projects for Assistance in Transition from Homelessness (PATH), and (4) the Treatment for Homeless Persons Program. The initial research task was aimed at developing an understanding of the basic operations of these four DHHS homeless-serving programs, with a particular focus on each programs performance measurement systems. Project staff conducted in-person discussions in December 2001 and January 2002 with programmatic, budget, and policy staff in the agencies that oversee these four DHHS programs. Appendix A provides a copy of the discussion guides used in conducting interviews with agency officials.

Background and Study Objectives

Recent studies suggest that homelessness is a problem that afflicts many adults and children in the nation and can have a broad range of short- and long-term negative consequences. It is estimated that up to 600,000 people in the United States are homeless each night.(4) A recent study by the U.S. Department of Health and Human Services, Ending Chronic Homelessness:  Strategies for Action, indicated that each year approximately one percent of the U.S. population  two to three million individuals  experiences a night of homelessness that puts them in contact with a homeless assistance provider.(5) The poor are particularly vulnerable to experiencing both short- and long-term periods of homelessness, with between four and six percent of the poor experience homelessness annually. This study also notes that the circumstances leading to homelessness are varied and that research conducted since the late 1980s shows that interactions among the supply of affordable housing, poverty, and disability account for most of the precipitating factors.

For those falling into homelessness  especially chronic homelessness  there can be a broad range of adverse effects. Without a stable residence, homeless individuals are faced daily with having to meet even their most basic and immediate needs to survive. Homelessness may threaten family integrity by exacerbating problems such as parental stress, emotional and health problems, alcohol and drug abuse, and family violence. Compared to families and individuals living in stable housing, those who are homelessness are more likely to be exposed to violence, illegal activity, illness, accident, malnutrition, depression, anxiety, and social isolation. Homelessness can make it very difficult to secure work, and even when a homeless individual is employed, the conditions of homelessness may jeopardize the ability to hold onto the job. Personal cleanliness, appropriate clothing, punctuality, and the energy to meet job expectations may all be difficult to maintain under unstable living arrangements.

Given the consequences of homelessness, effective intervention is important to prevent chronic or cyclical homelessness from occurring. For families and individuals, becoming homeless is a process that offers numerous points at which intervention and appropriate service might prevent the crisis that results in homelessness or mitigate its detrimental effects. Past studies  such as a recent Report to Congress⁽⁶⁾  have identified a broad continuum of services needed by homeless individuals to escape homelessness  particularly, housing assistance, health care services (including mental health care services and substance abuse treatment/counseling), employment and training services, and a range of support services (such as transportation, clothing, and food assistance). These services may help homeless individuals to overcome a current homeless episode or help individuals to avoid falling into a pattern of chronic homelessness.

In developing programs to address the needs of the homeless, it is important to specify clearly the program goals and objectives to guide implementation of program activities, as well as a set of performance measures to facilitate documentation and analysis of the effectiveness of program interventions. This study  conducted under a task order contract to the U.S. Department of Health and Human Services  explores the feasibility of developing a core set of performance measures for DHHS programs that focus on homelessness. It has two main objectives: (1) determine the feasibility of producing a core set of performance measures that describe accomplishments (as reflected in process and outcome measures) of the homeless-specific service programs of DHHS; and (2) determine if a core set of performance measures for homeless-specific programs in DHHS could be generated by other mainstream service programs supported by DHHS to assist low income or disabled persons.⁽⁷⁾

A key focus of the study is on enhancing performance measurement across four homeless-serving programs administered by DHHS: (1) Programs for Runaway and Homeless Youth (RHY), (2) the Health Care for the Homeless (HCH) Program, (3) Projects for Assistance in Transition from Homelessness (PATH), and (4) the Treatment for Homeless Persons Program.⁽⁸⁾ This study builds upon the process and outcome measures that are already generated as part of the homeless registry/homeless administrative data system (HADS) systems. In addition, this project deals with an important government management requirement that has affected agencies and programs for the past several years: the Government Performance and Results Act (GPRA), which requires government agencies to develop measures of performance, set standards for the measures, and track their accomplishments in meeting the standards.⁽⁹⁾

Study Methodology and Structure of the Report

This study mainly involved interviews with program officials knowledgeable about the four homeless-serving programs that were the main focus of this study, along with review of existing documentation. Interviews were conducted both by telephone and in-person. In addition, the research team conducted telephone interviews with program officials at four mainstream programs that are profiled in Chapter 5 of this report  Medicaid, the Substance Abuse Prevention and Treatment Program (SAPT), Head Start, and the Health Care Clusters programs. Project staff also reviewed documents and interviewed program officials that operated homeless administrative data systems (HADS) or homeless registry systems in several localities across the country. Each chapter includes additional details about specific data collection methods undertaken and the appendices to this report contain discussion guides used during interviews.

The remainder of this report is divided into four chapters. Chapter 2 of this report synthesizes the results of interviews with administrators and a review of relevant program documentation at the four DHHS homeless programs that are the focus of this study: (1) Programs for Runaway and Homeless Youth (RHY) Program, (2) the Health Care for the Homeless (HCH) Program, (3) Projects for Assistance in Transition from Homelessness (PATH), and (4) the Treatment for Homeless Persons Program. The chapter provides an overview of the basic operations of these four DHHS homeless-serving programs, with a particular focus on each programs performance measure systems and prospects for enhanced tracking of homeless individuals served.

Chapter 3 synthesizes the results of interviews with administrators and a review of relevant background documentation on the operations of homeless administrative data systems (HADS) in five localities (1) New York City, NY; (2) Madison, WI; (3) Kansas City, KS; (4) Columbus, OH; and (5) Honolulu, HI. This chapter provides an overview of the operations of these five HADS and analyzes the potential that the data collection methods and measures employed in these systems might have for enhancing performance measurement in the DHHS homeless-serving programs.

Chapter 4 identifies a potential core set of performance measures that could be common across homeless-serving programs of DHHS. The measures  including both process and outcome measures  suggested in this chapter are intended to enhance DHHS tracking of services and outcomes for homeless individuals served in DHHS homeless-serving and non-homeless-serving programs. This chapter includes discussion of several of the constraints in creating core performance measures, identifies a potential core set of homeless measures, and examines the technical implications for incorporating such measures into the current performance reporting approaches utilized by DHHS.

Chapter 5 assesses the potential applicability of the core set of suggested measures to four mainstream DHHS programs that serve both homeless and non-homeless populations: (1) Medicaid, (2) the Health Centers Cluster, (3) the Substance Abuse Prevention and Treatment (SAPT) Block Grant, and (4) Head Start. A key focus of this chapter is on assessing the capability and willingness of other mainstream DHHS programs to collect basic data relating to the number and types of homeless individual served, and moving beyond counts of homeless individuals served to adopting other suggested core performance measures.

Endnotes

(4) An estimate provided by the U.S. Department of Health and Human Services website (see http://www.hhs.gov/homeless/).

(5) U.S. Department of Health and Human Services, Ending Chronic Homelessness: Strategies for Action:  Report from the Secretarys Work Group on Ending Chronic Homelessness, March 2003, p. 5.

(6)J. Trutko, B. Barnow, S. Beck, S. Min, and K. Isbell, Employment and Training for Americas Homeless:  Final Report on the Job Training for the Homeless Demonstration  Report to Congress, Research and Evaluation Report Series 98-A, prepared for the Employment and Training Administration, U.S. Department of Labor, 1998.

(7) Initially, the study was also had a third objective  to determine if an index of chronic homelessness could be developed that helps both in treatment planning and documentation of program success  but during the project a DHHS advisory group developed such an index independent of this study.

(8) The Treatment for Homeless Persons Program was formerly referred to as the Addiction Treatment for Homeless Persons Program.

(9) The Government Performance and Results Act (GPRA) of 1993 seeks to shift the focus of government decision making and accountability away from a preoccupation with the activities that are undertaken  such as grants dispensed or inspections made  to a focus on the results of those activities, such as real gains in employability, safety, responsiveness, and program quality. Under GPRA, agencies are to develop multi-year strategic plans, annual performance plans, and annual performance reports.

Review of Reporting and Performance Measurement Approaches Among Four Homeless-Serving Programs Administered by the U.S. Department of Health and Human Services

Main Findings from Interviews with Agency Officials and Document Reviews

Overall, our interviews with agency officials at the four homeless-serving programs provided: (1) background information about each program, including information about key program components and client flow; (2) principal performance measures; (3) methods used to collect performance data; and (4) program officials views on potential measures that might be incorporated to enhance performance monitoring. In synthesizing the results from our discussions and review of background documentation on each project, we have attempted to the extent possible to provide cross-program comparisons at a fairly detailed level of key program features and, particularly, with respect to performance measurement used by each of the programs. Exhibit 2-1 provides a comparative analysis of some of the key programmatic features of the four homeless-serving programs. Below, we highlight several key findings that emerge from this program comparison.

Program funding, allocation, role of the federal/state governments, and number and types of agencies providing services vary substantially across programs. While all four of the programs serve homeless individuals as their target population, there are substantial cross-program differences that complicate efforts to develop and implement common measures of performance and systems for collecting data across the four programs. Some underlying programmatic differences are highlighted in the exhibit:

• Authorizing Legislation: Authorizing legislation for two of the four programs comes from the McKinney Act (PATH and HCH). Authorization for RHY dates back nearly three decades to Juvenile Justice and Delinquency Act of 1974, while authorization came only about two years ago (in 2001 by Congressional directive) for the Treatment for Homeless Persons Program.
• Budget, Funds Allocation, and Matching Requirements: FY 2002 funding runs from $9 million (Treatment for Homeless Persons) to in excess of $100 million ($116 million for HCH). Three of the four programs allocate funds competitively; one of the programs allocates funds to states by formula (PATH). However, even within the three programs using a competitive process to select grantees, the methods for allocating funds and selecting grantees are quite different and quite complex. For example, under RHYs Basic Center Program, 90 percent of funds are allocated to states based on the state population under age 18 in proportion to the national total. Regardless of the size of its youth population, the minimum allocation for a state is $100,000 ($45,000 for territories). Despite this initial allocation to states, the federal government runs a competitive grant process in which service providers (mostly local nonprofits and county agencies, though state agencies may also compete) submit grant applications. The applications are peer reviewed and awarded based on scores. If all the funding is not awarded within a state (i.e., all grant awards within a state do not add up to a states allocation), funds are re-allocated from that state to other states within the states region to fund other grant awards. By comparison, SAMHSA/CSAT issued Guidance for Applicants (GFA) for the Treatment for Homeless Persons Program for the first two rounds of grant awards. Nonprofit agencies submitted proposals and CSAT rated proposals and made awards to agencies based on how their proposals stacked up with others submitted. Finally, across the four programs, requirements for providing matching funds ranged from no match to providing $1 in non-federal funds for every $10 of federal funds to providing $1 in non-federal funds to every $3 of federal funds.
• Role of Federal versus State Governments: The federal government plays a significant role in all four of the programs  distributing funds to states (PATH) or competing grants and selecting grantees (in the case of the other three programs); providing oversight and collecting performance information; and providing technical assistance. In terms of state involvement, only under the PATH program among the four programs does the state play a significant role  distributing funds to local areas, providing technical assistance to local grantees, monitoring subgrantee performance, and submitting annual performance reports to the federal government.
• Number and Types of Grantees/Subgrantees: The number of grantees ranges from 50 grantees selected under the Treatment for Homeless Persons Program to about 640 under the three RHY programs. There is some overlap across programs in the types of local agencies receiving grant funds and providing direct services. All four of the programs rely to some extent upon nonprofit community-based organizations to recruit and deliver services (e.g., two-thirds of HCH grantees and about one-fourth of HCH grantees are CBOs). Community mental health centers account for nearly two-thirds of PATH grantees; CHCs represent about half of current HCH grantees; and RHY Basic Center Program grantees include a broad network of (about 400) youth shelters operated by public and nonprofit entities.

While there is a similar focus on homeless individuals across the four programs, there are differences in terms of the number and types of individuals served, definitions of enrollment, and duration of involvement in services. Not surprisingly, all four programs serve homeless individuals  though programs target different subpopulations of the homeless. The RHY programs target youth (both runaway and homeless), while the other three programs target services primarily on adult populations (though other family members are often also served). The HCH program funds initiatives that serve a broad range of homeless individuals (especially those unable to secure medical care by other means). The PATH and Treatment for Homeless Persons Programs serve a somewhat narrower subgroup of the homeless population than the other programs: the PATH program focuses on homeless individuals with serious mental illness; and the Treatment for Homeless Persons program targets homeless persons who have a substance abuse disorder, or both a diagnosable substance abuse disorder and co-occurring mental illness or emotional impairment.

The definition of enrollment and termination in the programs and duration of involvement in services all vary considerably by program. In a program such as PATH  which is considered to be a funding stream at the local (operational) level  it is often difficult to identify a point at which someone is enrolled or terminates from PATH. In a program such as HCH, a homeless individual becomes a participant when he/she receives clinical services at an HCH site. Length of participation in HCH is highly variable  it could range from a single visit to years of involvement. HCH program grantees would like to become the medical home for each individual until a point at which they are no longer homeless and can connect with another health care provider (to serve as the medical home). Much like any other private practice doctor, there is not generally a point in time in which an individual is terminated  rather, a case file is set up on the individual and at some point they simply do not show up (or may come in only sporadically for services).

Even within a program like RHY  which is composed of three program components  there is considerable variation in what constitutes enrollment and duration of involvement. For example, in RHYs Street Outreach Program (SOP)  a program designed to get youth off the street and into a safe situation (and linked to needed services)  involvement is very brief (often a single contact) and presents little opportunity for collecting information about the individual. In contrast, RHYs Transitional Living Program (TLP) provides residential care for up to 18 months under the program and a broad range of other services to move homeless youth toward self-sufficiency and independent living. RHYs third program component  Basic Center Program (BCP)  offers up to 15 days of emergency residential care, help with family reunification, and other services. Hence, BCPs involvement with homeless youth is longer and more intensive than SOP, but much shorter and less intensive than TLP.

Finally, of the four programs, enrollment in the Treatment for Homeless Persons Program appears to be most clearly defined. Homeless individuals are considered participants when the intake form (part of the Core Client Outcomes form) is completed on the individual (though there is no standardized time or point at which this form is to be completed by program sites). Involvement in the program is extended over a year or longer  with follow-up surveys being conducted with participants at six and 12 months after intake into the program.

While actual numbers of individuals served or participating are difficult (if not impossible) to compare because of varying definitions across programs, the sizes of programs appear quite different. For example, HCH (with 142 grantees nationwide) reports that about 500,000 persons were seen in CY 2000. Under its BCP program component (with a network of about 400 youth shelters nationwide providing services), the RHY program estimates that it helps 80,000 runaway and homeless youth each year. According to figures reported annually by states, the number of homeless individuals with serious mental illness who were PATH clients in FY 2000 was about 64,000 (though as noted earlier, because PATH is regarded as a funding stream rather than a distinct program, it is often difficult to isolate an individual as a participant or being served by PATH). Finally, through the first two rounds of funding, the 36 grantees funded under the Treatment for Homeless Persons Program anticipate serving about 7,700 individuals (over the three-year grant period).

Wide range of program services offered through the four programs. As shown earlier in Exhibit 2-1, despite their many differences, there is a fair degree of convergence in the goals of the four DHHS homeless-serving programs. All four of the programs are aimed at improving prospects for long-term self-sufficiency, promoting housing stability, and reducing the chances that participants will become chronically homeless. Each program has more specific goals that relate to the populations served and the original program intent for example, RHYs BCP component has as one of its goals family reunification (when appropriate); HCH aims to improve health care status of homeless individuals; PATH aims to engage participants in mental health care services and improve mental health status; and the Treatment for Homeless Persons Program aims at engaging participants in substance abuse treatment and reducing/eliminating substance abuse dependency.

Exhibit 2-1 (shown earlier) provides an overview of services delivered through the four programs. Common themes cutting across the programs include emphases on flexibility, providing community-based services, creating linkages across various types of homeless-serving agencies, tailoring services to each individuals needs (through assessment and case management), and providing a continuum of care to help break the cycle of homelessness. For example, the Treatment for Homeless Persons Program emphasizes linkages between substance abuse treatment, mental health, primary health, and housing assistance; HCH emphasizes a multidisciplinary approach to delivering care to homeless persons, combining aggressive street outreach, with integrated systems of primary care, mental health and substance abuse services, case management, and client advocacy. Of the four programs, the RHY program (in part, because it targets youth) provides perhaps the most unique mix of program services  and even within RHY, each program component provides a very distinctive blend of services (e.g., street outreach [the Street Outreach Program] versus emergency residential care [Basic Center Program] versus up to 18 months of residential living [Transitional Living Program]).

The four homeless programs feature substantially different approaches to performance measurement, collection of data, and evaluation. Given the variation in the structure of these four programs, it perhaps comes as no surprise that their approaches to information collection, performance measurement, and evaluation are quite different (see Exhibit 2-2). With respect to GPRA measures, three of the four programs have explicit measures; there are no GPRA measures specific to HCH. GPRA measures apply to the BPHCs Health Centers Cluster of programs as a whole, of which HCH program is part.⁽¹⁰⁾ The measures used for the three other programs range from process to outcome measures. The Treatment for Homeless Persons Program has outcome-oriented GPRA measures, as well as a data collection methodology (featuring intake and follow-up client surveys) designed to provide participant-level data necessary to produce the outcome data needed to meet reporting requirements. For example, the GPRA measures for adults⁽¹¹⁾ served by the Treatment for Homeless Persons Programs are the percent of service recipients who  (1) have no past month substance abuse; (2) have no or reduced alcohol or illegal drug consequences; (3) are permanently housed in the community; (4) are employed; (5) have no or reduced involvement with the criminal justice system; and (6) have good or improved health and mental health status. In contrast, the measures employed by PATH are process measures (rather than outcome-oriented): (1) percentage of agencies funded providing outreach services; (2) number of persons contacted, (3) of those contacted, percent enrolled in PATH. Of the three main GPRA measures used in the RHY program, just the first one is outcome-oriented: (1) maintain the proportion of youth living in safe and appropriate settings after exiting ACF-funded services; (2) increase the proportion of BCP and TLP youth receiving peer counseling through program services; and (3) increase the proportion of ACF-supported youth programs that are using community networking and outreach activities to strengthen services.

As displayed in Exhibit 2-2, how performance data are collected and the quality of the data collected also varies across the four programs. Three of the four programs have states (PATH) or grantees (HCH and RHY) submit aggregate data tables either annually or semi-annually. For example, under the PATH program, states submit 16 tables annually, including  (1) federal PATH funds allocated to the state, (2) total FTEs providing PATH supported services, (3) PATH providers by type of organization, (4) state/local matching funds, (5) PATH portion of local provider budgets, (6) PATH clients as a percentage of homeless clients in all services, (7) number of organizations providing PATH services by type of service and funding, (8) number and percent of PATH outreach contacts that eventually become enrolled in services, and (9) number and percent of PATH clients by: age, gender, race, principal diagnosis, dual diagnosis, veteran status, clients housing status, and length of time homeless. States send these tables via the Internet to the federal program office (HPB/CMHS), which abstracts data from each state to generate figures needed on each of the three GPRA measures. Data provided is aggregate (rather than at the participant-level) for PATH (as is the case for HCH and RHY).

The Treatment for Homeless Program (the newest of the four programs) takes an entirely different approach to collection of data than the other three programs. Each of the grantees collects participant-level data at three points of the clients involvement in the program (using standardized data collection form across all sites, referred to as the Core Client Outcomes form)  at intake, 6 months after intake, and 12 months after intake. This generates the data needed by the program to address the outcome-oriented GPRA measurement (e.g., percent of participants who have no past month substance abuse). In addition, it is possible on an individual participant basis to make comparisons on the various outcome measures between the time of intake and follow-up to determine pre/post change for each participant.

All four of the programs use (or are in the process of developing and implementing) some type of automated database for transmission of performance data to their federal administering agencies. In the case of PATH, HCH, and RHY standardized data tables are produced by each state (PATH) or grantee (HCH and RHY) and submitted via the Internet. CSAT has recently implemented a web-based application, which enables Treatment for Homeless Persons Program grantees to submit participant-level records via the Internet.

There are several other issues with regard to the collection of performance data that affect their appropriateness for GPRA reporting and evaluating program performance:

• The reliability and quality of the data collected and submitted to federal offices varies by program. For example, RHY is substantially revising and streamlining its data system (RHYMIS) in response to past problems with data completeness and quality. A recent report noted RHYMIS data are unreliable because of chronic low levels of grantee reporting (less than 50% of grantees submitted reports for all four quarters in FY 1999); also, many youth served by RHY centers are not counted as admitted to services in RHYMIS because services are funded by non-federal sources.
• The HCH program is embedded in the Health Cluster of programs at BPHC (and is a relatively small program despite its in excess of $100 million funding, when compared to the Community Health Centers funding). Annual reports submitted by grantees receiving more than one Health Cluster source of funds combine HCH participants with participants of other Health Cluster programs and, hence, it is not possible (except on several of the tables provided by grantees) to produce disaggregated counts for HCH participants.
• Sophistication with data collection and reporting varies considerably across and within programs. For example, RHYs BCP program funds about 400 youth shelters. A recent RHY report indicated that funded agencies range from large, multi-service agencies with fairly sophisticated data collection capabilities to small single-service agencies just beginning to use MIS technology to track service delivery.
• Intensity and duration of participant involvement in programs ranges considerable across and within programs. As noted earlier, short or episodic involvement of participants in programs (such as that of some participants of the RHY and HCH programs) limit opportunities for in-depth collection of data from participants.

Program Use Data for a Variety of Purposes. Data collected by the four programs are used for a broad set of purposes, particularly reporting to Congress and others about the program, budgeting purposes, deciding on how to allocate funds to grantees, and to support evaluations of the programs and technical assistance efforts. Programs do not use the data at this time for performance rewards  though in some cases, the data has an effect on which grantees are funded in future rounds. As described by agency officials, the following are the main ways in which data currently collects are being used:

• HCH: Data are used mainly to report to the Department and Congress on whom the program serves and the types of services provided. No rewards or sanctions result directly from data collected  though technical assistance may be provided for those with poor performance. Some funding decisions are based, in part, on trends in users over several years time.
• PATH: The data collected during the grant application process and through the annual reporting system are used to generate cross-state data tables and U.S. totals to analyze program and participant characteristics. These data are used to report to Congress on the program and to generate data on the three GRPA measures. The federal office carefully reviews annual reports and grant applications  providing comments where appropriate to states.
• Treatment for Homeless Persons Program: Though the program just recently started, data collected will ultimately be used to report to Congress on the performance of the program. No performance bonuses are planned.
• RHY: Data collected from grantees is used by the Family and Youth Services Bureau (FYSB) to report annually on the program. Data are not used to distribute performance awards; data collected may be used to guide decisions in issuing future grants to existing or past grantees.

Implications and Conclusions for Development of Common Performance Measures

Initial discussions with agency officials at the four homeless-serving programs and review of readily available program documentation suggested that despite having a common focus on serving homeless individuals, the four programs that are the focus of this study have many differences. Upon closer examination of the programs, the differences appear to be greater than the similarities for example, the four programs serve different subpopulations of the homeless, providing a different range of services over varying lengths of participant involvement, to achieve often different results. As might be expected given these programmatic differences, approaches to measuring and reporting on program performance and the problems associated with collecting high-quality data are also quite different. In particular, sites vary substantially across the following dimensions:

• Actual GPRA measures used  ranging from process to outcome-oriented;
• The specific data items collected and the extent to which pre/post outcome data are collected;
• Whether data are maintained and submitted to the federal office in aggregate or at the participant-level;
• Reliability and completeness of data provided; and
• Whether new performance measures and data systems are currently being designed and/or implemented.

The implication of these differences is that it will be a difficult and delicate task to come up with a common set of performance measures across the four programs, which are also applicable to other DHHS programs serving homeless individuals. In addition, while federal agency officials are very willing to discuss their programs and share their knowledge of how they approach data collection and reporting, their willingness and ability to undertake change (e.g., potentially incorporating new, more outcome-oriented GPRA measures) is uncertain. From our discussions, it appears that changes in how programs collect data and report on performance will require substantial efforts on the part of agency officials and programs. For example, with regard to RHY  which is currently involved in an effort to implement a streamlined data system  it would not only require change at the federal administering agency, but how over 600 grantee organizations collect and manage data.

In the next chapter of this report, we examine the potential relevance of homeless administrative data systems (HADS) for enhancing data collection and performance measurement in DHHS homeless-serving programs. Chapter 4 then returns to the main focus of this study  examining the potential for implementing a set of common performance measures across these four homeless-serving DHHS programs.

Endnotes

(10) HCH is clustered with several other programs, including Community Health Centers [CHCs], Migrant Health Centers, Health Services for Residents of Public Housing, and other community-based health programs.

(11) As shown in Exhibit 2-2, GPRA measures are slightly different for youth.

Analysis of Measures Derived from Homeless Administrative Data Systems (HADS)

This chapter synthesizes the results of interviews with administrators and a review of relevant background documentation on the operations of homeless administrative data systems (HADS) in five localitiesВ  (1) New York City, NY; (2) Madison, WI; (3) Kansas City, KS; (4) Columbus, OH; and (5) Honolulu, HI. This study activity was focused on (1) collection of background information about homeless registry approaches or HADS and (2) analysis of the potential that the data collection methods and measures employed in these systems might have for enhancing performance measurement in the DHHS homeless-serving programs that are the overall focus of this project.

With input from DHHS, we selected five HADS (in New York City, Madison, Columbus, Kansas City, and Honolulu) for study. In the Summer 2002, we interviewed (by telephone) system administrators about the operations of each of the five HADS (see Appendix B for a copy of the discussion guide). Agency officials were very cooperative in terms of sharing both their knowledge of and perspectives on their systems (including some of the problems and limitations of such systems), as well as in providing background documentation about main features and data elements included in their systems. In addition, in several instances, we were able to view the HADS via Internet websites provided by the sites. Project staff also conducted a follow-up site visit in the Summer 2002 to New York Citys Department of Homeless Services to interview staff in greater depth and obtain additional background information on the operation of HADS. The New York system was selected for a site visit because of: (1) the very large number of homeless individuals on which the system maintains information (e.g., estimated at nearly one million homeless individuals); (2) the long time that the system has been operational (since the mid-1980s); (3) the systems focus exclusively on tracking homeless individuals and families; and (4) the fact that the Department is currently making a transition to a new system that will use the latest in hardware and software technologies.

Based on the results of our interviews and review of background documentation, project staff analyzed key features of the selected HADS and the implications of these systems for enhancing performance measurement in DHHS homeless-serving programs. Appendix C provides copies of some background documentation on key features of these systems.

Main Findings from Interviews with HADS Administrators and Reviews of Background Documentation on HADS

Exhibit 3-1 provides a comparison of key HADS features and performance measures (as of the Summer 2002, when our interviews were conducted) in the five local sites included in this study. Below, we highlight key findings that emerge from our examination of these five HADS.

The HADS system in New York has been operational since 1986, while the other four have been designed and implemented during the past decade; all five systems are either in the process of being upgraded to use the most recent technology or were recently developed using state-of-the-art technology. As shown in Exhibit 3-1, of the five localities examined, New York Citys system is the oldest  originating in the mid-1980s. At the time of our visit to NYC, the Department of Homeless Services was pilot testing a new HADS that included basically the same data elements as the former system, but featured the latest technology in terms of hardware and software components. For example, the system is being developed using an ORACLE platform, which will enable emergency and transitional facilities located across New York City to input data directly into the system via T-1 lines. The other four HADS have all been developed and implemented within the past 10 years. The system used in Kansas City  MAACLink  originated in 1994, though during the past year its sponsoring agency (the Mid-America Assistance Coalition) has spent about $200,000 enhancing the software and other operational aspects of the system. The system utilized in Hawaii  referred to as the State Homeless Shelter Stipend Database  was initially implemented also in 1994, but is currently being substantially revised and upgraded to become a web-based application (with the new system expected to become operational by the end of 2002). The homeless agencies in Madison (WI) and Columbus (OH) have implemented the ServicePoint system, a web-based system designed by Bowman Internet Systems. The Community Shelter Board (in Columbus) implemented the system in 2000 (though 10 years of previous data was subsequently uploaded to the system), while the Bureau of Housing (in Madison) implemented the ServicePoint system in 2001. The ServicePoint system is a web-based application, with data entered at remote service sites (i.e., homeless-serving agencies in the Madison and Columbus areas) and sent electronically over the Internet for storage at secure file servers located on the premises of Bowman Internet Systems (located in Louisiana).

HADS tend to be system-wide  some cutting across a large number of partners  which avoids focusing narrowly on programs (e.g., ?silos). Several of the HADS are very large in terms of the number of partnering organizations and programs that are linked via the systems. The largest in terms of number of partnering agencies  that is, agencies providing data for entry into the HADS  is the MAACLink system in Kansas City. A total of 227 partnering agencies from a five-county area surrounding Kansas City contribute data to the MAACLink system (either on-line or by submitting hardcopy forms for entry into the system by MAAC). Partners include some homeless-serving agencies, but also other agencies serving low-income individuals or families in the Kansas City area. Nearly 60 percent of the MAACLink partnering agencies (135 agencies) are connected on-line to the HADS. The ServicePoint systems used in Madison and Columbus also have a large and diversified pool of partnering agencies. The system maintained in Madison has 84 partnering agencies from across Wisconsin contributing data. Partners include a broad range of agencies that target services on homeless individuals and others at-risk of homelessness  including emergency shelters, transitional/supportive housing agencies, local housing authorities, domestic violence service providers, faith-based organizations, and tribal agencies. Program officials (at the Bureau of Housing in Madison) anticipate that the number of partnering agencies will grow to about 225 by the Summer 2004. Some potential new partners include food banks, clothing pantries, and other agencies that provide support services needed by homeless and other low-income households. The ServicePoint system in Columbus brings together 28 partners, but partnering agencies are more narrowly focused (than in Madison or Kansas City) to include only homeless-serving agencies (such as emergency shelters, homeless prevention programs, resource centers, and housing search assistance agencies). The New York City system is focused exclusively on collecting data on homeless individuals and families served within emergency and transitional housing funded by the NYC Department of Homeless Services. The only other partner providing data for the HADS is the NYC Human Resources Administration, which provides data (merged into the HADS) to indicate whether homeless individuals/families included in the HADS are also public assistance recipients. The Housing and Community Development Corporation in Hawaii partners on the HADS with agencies it directly funds to provide street outreach, emergency shelter, and transitional shelter for homeless individuals and families.

Some HADS have accumulated substantial numbers of records on homeless and other types of disadvantaged/low-income households. These systems demonstrate that it is possible to collect and share data across a broad range of program. The NYC HADS is one of the largest (if not the largest) in the country  having accumulated an estimated 800,000 to 900,000 records on homeless individuals served by emergency and transitional housing facilities in the New York City since the inception of the system in 1986. The system includes only homeless individuals and families served in NYCs shelter system. The system creates a single case record for each individual, which displays all episodes of receipt of housing assistance through emergency or transitional facilities over the last 16 years (though there are some duplicate records because people use aliases or fail to provide accurate identifying information). On an average day in June 2002, there were 7,903 families in temporary housing and 7,557 single adults in shelters in New York City (a total of over 30,000 individuals in homeless facilities)⁽¹²⁾  all of which are entered into the data system. The MAACLink maintained in Kansas City also has a very large number of records in its system  an estimated 450,000 individuals (an estimated 112,000 new records were entered into the system in 2001). However, the 227 partnering agencies (many of which provide services for a wide range of low-income and disadvantaged individuals) can enter data into the system on anyone using their services  and hence, only a relatively small percentage of those entered into the system are or have been homeless.

The other HADS are much smaller in relative terms (when compared to Kansas City and NYC), but still contain significant numbers of records  about 40,000 individuals in the Madison HADS (the vast majority of which are either homeless or individuals at-risk of homelessness); about 54,000 individuals in the Columbus system (which includes mostly homeless or those at-risk of homelessness); and about 100,000 individuals in the Hawaii system (which is limited to homeless individuals in emergency or transitional shelters or contacted as a result of street outreach efforts; however, new records on individuals served each year are created, and so, there are many individuals with duplicate records from year to year).

HADS systems are not used principally for measuring program performance or outcomes  though have the capability to provide analyses of length of stay. The HADS principally serve as registry systems that facilitate tracking of program participant characteristics, services received, length of stay, and movement within emergency and transitional housing facilities. The systems can also be useful in avoiding duplication of services, reducing fraud and abuse, and facilitating payment to vendors. Exhibit 3-1 (shown earlier) provides a general overview of the key data elements collected in each of the five systems; Appendix C contains additional documentation on data elements from several of the sites that provided hardcopy forms and additional background on their data systems. All of the HADS in our survey collect client identifiers (e.g., name, Social Security Number, and address, if available), as well as a basic set of demographic characteristics. Among the core of basic demographic features being collected in most sites are gender, age, race/ethnicity, marital status, and veteran status. HADS vary in terms of other types of client characteristics data collected. Some example of other types of background information collected on the individual include educational attainment, income and income sources, employment status, living arrangement, household size, health status, substance abuse problems, mental health problems, and other special needs. Several of the systems (New York, Madison, and Columbus) collect information about reasons for homelessness. The ServicePoint system used in Madison collects what appears to be the most information concerning the individuals housing/homeless situation prior to entry into the program  including current living situation, homeless status, reasons for homelessness, date the individual became homeless, whether this episode is the first time the individual has been homeless, and reason the individual left his/her prior living situation.

The HADS in each of the five localities track some type of service data and length of stay in shelter facilities  but the types tracked and the extent to which data are analyzed varied considerably across sites. The New York City HADS collects data on the referral date, the name of the emergency or transitional facility to which the individual/family is referred, the room number, the date of entry and exit from the shelter facility, and total days housed within the facility. The New York City HADS provides the Department of Homeless Services with the data needed to validate invoices submitted by shelters for payment for specific days of shelter use for each individual/family. In addition, the system in New York City enables the Department to analyze characteristics of individuals served, track individuals into and out of shelter facilities, and monitor shelter capacity and facilitate placement of individuals/families into appropriate vacant units. Analyses by Kuhn and Culhane⁽¹³⁾ of data from New Yorks HADS illustrate the types of outcome analyses that are possible with HADS data and some of the limitations to use of such data. For example, Kuhn and Culhane were able to analyze length of stay for users of homeless shelters for over 70,000 homeless individuals between 1988 and 1995. Using available data, the researchers identified three distinct groups of users  transitionally homeless (81 percent), episodically homeless (9 percent), and chronically homeless (10 percent).⁽¹⁴⁾ For the overall population and each of these three groups, the researchers analyzed: average number of episodes of homelessness, average number of days of homelessness, average days per episode, and total and percentage of client days in shelter. The background characteristics collected at the time each individual entered the NYC shelter system enabled Kuhn and Culhane to analyze HADS data overall and for each of these three groups across the following characteristics of shelter users: age, race/ethnicity, gender, and self-reported disabilities (limited to mental illness, medical problems, and substance abuse problems). The researchers concluded that The chronically homeless, who account for 10 percent of the shelter users, tend to be older, non-white, and to have higher levels of mental health, substance abuse, and medical problems.  Despite their relatively small numbers, the chronically homeless consume half of the total shelter days. The authors note that their study is limited by its reliance upon administrative data for recording periods of homelessness and for measuring characteristics of shelter users. For example, they point out data in the HADS in New York City on mental health, medical, and substance abuse problems are self-reported (and hence, may lack reliability) and that periods of street homelessness are not captured. In addition, the number of background variables collected on each individual is limited to just a few demographic variables, and outcome measurement is limited to analysis of length of stay and whether there are multiple episodes/readmissions to the shelter system (rather than, for example, whether an individual secures and keeps permanent housing, is able to find and keep a well-paying job, is able to achieve additional education qualifications, and is able to overcome substance abuse problems).

The Shelter Stipend Database in Hawaii, which was being substantially revamped at the time of our interview in July 2002 (but was expected to be operational by late 2002/early 2003), offers excellent opportunities for outcome analyses. This is because the system employs an exit form (see Appendix C for a copy of this form and others used in Hawaii), which captures data on a number of important outcomes (some of which may permit pre/post comparisons). Specific analyses that should be possible using the exit information to be collected as part of the upgraded system in Hawaii include the following:

• length of stay in the shelter facility (i.e., days between the date of entry into program to date of exit from the program);
• number of individuals within the family who left and remained at the shelter at the time the household head left the shelter;
• destination to which the individual/family was going at the time of exit, including: permanent housing (such as rental housing, public housing, a Section 8 unit, or homeownership), moved in with family, transitional housing, emergency shelter, drug treatment, unsheltered situation (e.g., street, park), hospice/care home, medical or psychiatric hospital, prison/jail, or destination unknown;
• reason for exit, including transitioned successfully, exited voluntarily, non-renewal of lease, left before completing the program, reached maximum time allowed in program, evicted, completed program services, needs could not be met by the program, disagreement with rules/person leading to termination from facility, arrested/left for prison, left for hospital, deceased, or unknown/disappeared;
• resources used at exit, including: public housing, Section 8, grant, loan, clients savings, financial support from friends/family, Hawaiian Homelands funding, no resources, and unknown;
• geographic location at the time of exit, including remained in Hawaii (specific island identified), left for mainland, and unknown;
• monthly household income by major source at the time of exit (i.e., sources include work, TANF, SSI, SSDI, retirement/pension, child support, workers compensation, unemployment benefits, Medicaid/Medicare, food stamps, financial help from family/friends, other, and unknown); and
• support services received during the time in the project, including outreach, case management, life skills, alcohol or drug abuse services, mental health services, HIV/AIDS-related services, other health care services, education, housing placement, employment assistance, child care transportation, legal, other, and unknown.

Using the data collected in Hawaii, for example, it should be possible to make some comparisons between conditions of the household at the time of entry with conditions at the time of exit, for example, in terms of total household income and income sources, and living situation just prior to entering the shelter system with the destination to which the household was going at the time of exit.

The Kansas City system enables partnering agencies to keep track of the services being provided by other agencies  and so helps to eliminate duplication of services. The system was also designed for a very specific purpose  to enable partnering agencies to automatically process emergency utility vouchers. The system is programmed so that each partner can automatically determine if an individual meets eligibility guidelines for utility vouchers and accounts for issuance of each voucher by the partnering agency. The system also enables partnering agencies to calculate family budget, which also enables the agency to assess whether an individual is likely eligible for other types of assistance, including food stamps, energy assistance, and TANF. The Service Point systems in Madison and Columbus provide the sponsoring agencies with a wide array of reports for analysis purposes. Specific reports are available to analyze the number of homeless individuals served, participant characteristics, service needs, types of shelter facilities used, other support services provided, and length of stay.

Range of implementation challenges reported  particularly with regard to training system users to make full use of system features. Exhibit 3-1 highlights a variety of problems that agencies have encountered both in establishing their HADS and ensuring that systems are used appropriately by partnering agencies responsible for providing much of the data entered into the systems. Some of the implementation issues reported by agency officials we interviewed include the following:

• Developing a new system requires substantial staff effort in terms of programming and pilot testing the new application (New York).
• Training staff on how to appropriately and effectively use the system can be a huge issue, and because of staff turnover, there is a need for ongoing training. Partnering agencies often lack the technological capacity and know-how to operate systems without substantial training (Madison and Columbus).
• Federal reporting requirements vary substantially from agency to agency (particularly between DHHS and HUD), which can complicate ways in which data elements and reports are structured within data systems (Madison).
• Sharing of sensitive data across partnering agencies can be complicated and may require special programming so that access to such data can be limited to only certain partners and staff within agencies  for example, agencies serving individuals with domestic violence issues can be reluctant to share data that is available to other agencies (Kansas City and Hawaii).
• Quality controlling data can be an issue when data are being collected for the same individual by different agencies (Columbus).
• There can be problems in convincing partnering agencies to utilize standard system forms (Hawaii).
• Standard report formats may be inadequate for the specific reporting needs or information requirements of partnering agencies (Hawaii).

Implications of HADS for Development of Homeless Performance Measures

Overall, while the HADS reviewed for this report provide some useful measures of program inputs and process, they do not provide a set of measures of program outcomes or performance (with the possible exception of length of stay) that are readily adaptable to the DHHS homeless-serving programs that are the focus of our overall study. There are, however, some interesting implications that can be drawn from HADS for developing performance measures for DHHS homeless-serving programs and the systems capable of maintaining data that might be collected as part of such systems.

With regard to measures of homelessness several of the systems we reviewed do collect data on duration of episodes of receipt of homeless services (i.e., length of stay in emergency shelters and transitional facilities). Such a measure is particularly helpful in understanding frequency and total duration of homeless individuals receipt of assistance (e.g., duration of each spell of use of emergency shelters). Such data would be particularly helpful in understanding the extent of chronic homelessness and types of individuals most likely to have frequent and lengthy stays in emergency or transitional facilities. This points to the need to collect client-level data on service utilization, which includes dates that services begin and end so that it is possible to examine duration and intensity of services received, as well as multiple patterns of service use (i.e., multiple episodes of shelter use). The HADS also show that it is possible to collect detailed background characteristics on homeless (and other disadvantaged) individuals served, and especially in the case of Hawaiis HADS, to collect data at the time of entry and exit from homeless-serving programs to support pre/post analysis of participant outcomes.

The HADS systems also clearly demonstrate that it is possible to collect data on a core set of data items on homeless individuals receiving services from a substantial number of local homeless and other human services agencies. The systems demonstrate that it is possible to amass such data over a considerable period of time (15 years and longer) for a substantial number of homeless individuals (e.g., hundreds of thousands). In addition, the systems also demonstrate that very large networks of partnering agencies (in excess of 200 agencies) can collaborate on the development and implementation of data systems to track homeless and other types of disadvantaged individuals. Rapid technological advances in recent years  particularly the ability to input and retrieve data at remote service locations  have facilitated the expansion of such systems and made it possible for a wide variety of human service agencies (in some instances offering substantially different types of services) to share data on the same group of homeless and other disadvantaged individuals. Such sharing of data helps to facilitate inter-agency referrals, can contribute to reduction in duplication of services (e.g., reducing the chance that the same individual may receive utility or food vouchers for the same period from two different local agencies), help agencies to track homeless individuals/families over an extended period, and facilitate reporting of program service levels and results to state and federal agencies. Hence, while demonstrating the feasibility of implementing large systems to collect data on homeless individuals and linking substantial numbers of partnering agencies to collect such data, the HADS that we have reviewed for this study do not suggest a comprehensive list of performance measures that could be applied to DHHS homeless-serving programs. However, if a set of common measures were developed, the implementation experiences of the HADS would be helpful in terms of the lessons suggested for successful implementation of automated systems to maintain such data.

Endnotes

(12) The total of 30,000 homeless individuals includes single homeless adults in emergency shelters and families (composed of both adults and children) in temporary shelters.

(13) Randal Kuhn and Dennis P. Culhane, Applying Cluster Analysis to Test a Typology of Homelessness by Pattern of Shelter Utilization: Results form the Analysis of Administrative Data, American Journal of Community Psychology, Vol. 26, No. 2, 1998.

(14) Using cluster analysis on a sample of 73,263 total homeless individuals, Kuhn and Culhane found that the chronic cluster represented clients with a lone episode to six episodes with stay lengths from 371 to 1095 days over a three-year period; the episodic cluster represented clients with 3 to 14 stays over a three-year period, with stay length ranging from 1 to 895 days; and the transitional cluster included all others in the sample (with fewer spells and/or durations of spell length) that did not fall into the other two clusters.

Potential Core Performance Measures for Homeless-Specific Service Programs

This chapter identifies a potential core set of performance measures that could be common across homeless-serving programs of DHHS. The measuresВ  including both process and outcome measuresВ  suggested in this chapter are intended to enhance DHHS tracking of services and outcomes for homeless individuals served in DHHS homeless-serving and non-homeless-serving programs. This chapter includes the following sections: (a) discussion of several of the constraints in creating core performance measures; and (b) identification of a potential core set of homeless measures and discussion of technical implications for incorporating such measures into the current performance reporting approaches utilized by DHHS.

Considerations and Constraints on Developing a Common Set of Performance Measures

Using the material and analyses conducted under earlier study tasks, the focus of this chapter is on offering a set of suggested performance measures that could be common and useful across homeless-serving programs of DHHS. In our presentation and analysis of these measures, we have attempted to differentiate between performance measures that are possible from current reporting approaches and those derived from HADS operations, operations needed to collect and aggregate the data, and quality and uses of the data. In developing these measures, we took into consideration the following important factors:

• Extent currently collected. Items that are already collected by more programs have the advantages of already being highly regarded and contributing the least resistance for inclusion in a uniform system.
• Ease of collection. For items not universally collected, the ease at which an item can be collected is of interest. We are concerned with initial costs to establish the collection system as well as ongoing costs.
• Relationship to outcome and process measures of interest. In some instances, proxy measures for the measures of interest must be used because the proxies are preferable on criteria such as ease of collection and extent currently used.

In proposing a set of core performance measures for the four homeless-serving programs that are the focus of this study, the findings from our earlier review of each program and its current performance measurement system catalogue constraints for development of a common set of performance measures that cut across the programs. Perhaps most important, our earlier analysis of the four homeless-serving programs indicated that there are substantial cross-program differences that complicate efforts to develop similar performance measures and systems for collecting data (see Chapter 2 for more detailed discussion of cross-program differences and for a chart comparing the four homeless-serving programs). For example:

• Programs target different subpopulations of homeless individuals. For example, the RHY programs target youth (both runaway and homeless), while the other three programs target services primarily on adult populations (though other family members are often also served). While the HCH program funds initiatives that serve a broad range of homeless individuals (especially those unable to secure medical care by other means), the PATH program focuses on homeless individuals with serious mental illness; and the Treatment for Homeless Persons program targets homeless persons who have a substance abuse disorder, or both a diagnosable substance abuse disorder and co-occurring mental illness or emotional impairment.
• The definition of enrollment and termination in the programs and duration of involvement in services all vary considerably by program. For example, in a program such as PATH  which is considered to be a funding stream at the local operational level  it is often difficult to identify a point at which someone is enrolled or terminates from PATH. In a program such as HCH, a homeless individual becomes a participant when he/she receives clinical services at a HCH site. Length of participation in HCH is highly variable  it could range from a single visit to years of involvement. Finally, of the four programs, enrollment in the Treatment and Homeless Persons Program appears to be most clearly defined. Homeless individuals are considered participants when the intake form (part of the Core Client Outcomes form) is completed on the individual (though there is no standardized time or point at which this form is to be completed at sites).
• Numbers of homeless individuals served are quite different across the four programs. While actual numbers of individuals served or participating are difficult (if not impossible) to compare because of varying definitions across programs, the sizes of programs appear quite different. For example, HCH (with 142 grantees nationwide) reports that about 500,000 persons were seen in CY 2000. This compares with the RHY program estimates that it helps 80,000 runaway and homeless youth each year and estimates that PATH served (in FY 2000) about 64,000 homeless individuals with serious mental illness.
• Types of program services vary considerably across programs. Common themes cutting across the programs include emphases on flexibility, providing community-based services, creating linkages across various types of homeless-serving agencies, tailoring services to individuals needs (through assessment and case management), and providing a continuum of care to help break the cycle of homelessness. However, the specific services provided are quite different. For example, the Treatment for Homeless Persons Program emphasizes linkages between substance abuse treatment, mental health, primary health, and housing assistance; HCH emphasizes a multidisciplinary approach to delivering care to homeless persons, combining aggressive street outreach, with integrated systems of primary care, mental health and substance abuse services, case management, and client advocacy. Of the four programs, the RHY program (in part, because it targets youth) provides perhaps the most unique mix of program services  and even within RHY, each program component provides a very distinctive blend of services (e.g., street outreach [the Street Outreach Program] versus emergency residential care [Basic Center Program] versus up to 18 months of residential living [Transitional Living Program]).

Given the variation in the structure of these four programs, it is not surprising that the four homeless-serving programs have adopted quite different approaches to information collection, performance measurement, and evaluation (see Exhibit 2-2 earlier for specific measures used by each program). With respect to GPRA measures, three of the four programs have explicit measures; there are no GPRA measures specific to HCH. GPRA measures apply to the BPHCs Health Centers Cluster of programs as a whole, of which HCH program is part.⁽¹⁵⁾ The measures used for the three other programs include both process and outcome measures. The Treatment for Homeless Persons Program has outcome-oriented GPRA measures, as well as a data collection methodology (featuring intake and follow-up client surveys) designed to provide participant-level data necessary to produce the outcome data needed to meet reporting requirements. RHY and PATH employ mostly process-oriented GRPA measures.

Reliability and quality of data collected and submitted to federal offices varies by program. In addition, intensity and duration of participant involvement in the four homeless-serving programs ranges considerably across and within programs, with implications for performance measurement: short or episodic involvement (such as the involvement in some participants in RHY and HCH programs) limit opportunities for collection of data from participants.

In Chapter 2, we concluded that it would be both a difficult and delicate task to develop a common set of performance measures across the four homeless-serving programs. We noted that the willingness and ability of programs to undertake change (e.g., incorporate new outcome-oriented GPRA measures) is uncertain and the changes in how programs collect data and report on performance would require substantial efforts on the part of agency officials and programs. Hence, in specifying performance measures, it is important to be sensitive to the substantial cross program differences and the constraints that program administrators (at the federal, state, and local levels) face in making changes to how they collect and report on program performance.

Suggested Core Performance Measures

Despite the difficulties and constraints in developing a core set of performance measures, our review of the performance measurement systems in existence across the four programs also indicates potential for both enhancement and movement toward more outcome-oriented measures. For example, the general approach to performance measurement used within the Treatment for Homeless Persons Program  which features pre/post collection of participant-level data and outcome-oriented measures  provides a potential approach that could be applicable to the other three programs (as well as other non-homeless-serving programs operated by DHHS). In suggesting a potential set of core performance measures cutting across these four homeless-serving programs, it is important to consider where the four programs intersect with respect to program goals/objectives for the homeless individuals being served. From this commonality of goals arises the potential for a core set of measures (with the recognition, however, that each program will also likely require additional measures specific to differing objectives and service offerings). Of critical important to our efforts to suggest core measures, all four of the programs are aimed at (1) improving prospects for long-term self-sufficiency, (2) promoting housing stability, and (3) reducing the chances that individuals will become chronically homeless.⁽¹⁶⁾ In addition, the four programs (some more than others) also stress addressing mental and physical health concerns, as well as potential substance abuse issues.

Based on the common objectives of these four programs, we suggest a core set of process and outcome measure that could potentially be adapted for use by the four homeless-service programs (see Exhibit 4-1). We suggest selection of the four process measures, which track numbers of homeless individuals (1) contacted/outreached, (2) enrolled, (3) comprehensively assessed, and (4) receiving one or more core services. We then suggest selection of several outcome measures from among those grouped into the following areas: (1) housing status, (2) employment and earnings status, and (3) health status. In addition, we have suggested a several additional outcome measures that could be applied to homeless youth.

With regard to housing outcomes, we have identified three potential outcome measures intended to track (1) changes in an individuals housing situation along a continuum (from living on the street and in emergency shelters to securing in permanent housing), (2) whether the homeless individual secures permanent housing, and (3) days of homelessness during the preceding quarter (or month). It should be noted with regard to housing outcomes, that although the four homeless-serving programs focus primarily on other non-housing related goals and services (e.g., improving mental or physical health status, reducing/eliminating substance abuse, reuniting runaway youth with their families), that housing outcomes for homeless individuals are of paramount importance. Housing outcomes are appropriate to consider for programs focused on homelessness even when their primary goals may be focused on improving mental health status or physical health status.

Two earnings measures are identified  one that captures actual dollar amount of earnings during the past quarter (or month) and a second measure that captures whether an individuals earnings have improved. Two employment measures are also identified  one relating to whether the individual is engaged in work 30 or more hours per week and another that measures whether hours of work have increased. Three health-related measures are offered, focusing on use of drugs, improvement in physical health status, and improvement in mental health status. Finally, three measures are offered that are targeted exclusively on youth (though the other outcome measures would for the most part also be applicable to youth): (1) whether the youth is reunited with his/her family, (2) whether the homeless youth is attending school, and (3) whether the homeless youth graduates from high school or completes a GED.

A pre/post data collection approach is suggested with respect to obtaining needed performance data  for example, collecting data on housing, health, and substance abuse status of program participants at the time of intake/enrollment into a program and then periodically tracking status at different points during and after program services are provided (i.e., at termination/exit from the program and/or at 3, 6, or 12 months after enrollment). Collection of data on homeless individuals at the point of termination can be problematic because homeless individuals may abruptly stop coming for services. The transient nature of the homeless population can also present significant challenges to collecting data through follow-up surveys/interviews after homeless individuals have stopped participating in program services (e.g., at 12 months after enrollment).

Given difficulties of tracking homeless individuals over extended periods (and particularly after individuals termination from programs), the extent to which existing administrative data can be utilized could increase the proportion of individuals for which it is possible to gather outcome data (at a relatively low cost). Probably the most useful source in this regard is quarterly unemployment insurance (UI) wage record data, which can be matched by Social Security number (though releases are required and it may also be necessary to pay for the data). UI wage withholding data provides the opportunity to track earnings on a quarterly basis (from covered employers) and, for example, examine how earnings may change from quarter to quarter and potential effects of program involvement on workforce participation and economic self-sufficiency.

A second potential source of administrative data that may have some potential utility for tracking housing outcomes are HADS system maintained by many states and/or localities. As noted in Chapter 3, HADS systems are not used principally for measuring program performance or outcomes  though have the capability to provide analyses of length of stay. The HADS principally serve as registry systems that facilitate tracking of program participant characteristics, services received, length of stay, and movement within emergency and transitional housing facilities. Such systems may provide useful data for tracking use of emergency and transitional housing, as well as chronic homelessness  though are limited for purpose of determining housing status once an individual leaves emergency or transitional housing (i.e., on the street or in permanent housing).

Finally, in terms of tracking self-sufficiency outcomes, data sharing agreements with state and local welfare agencies may provide possibilities for tracking dependence on TANF, food stamps, general assistance, emergency assistance, and other human services programs.

Conclusions

The process and performance measures outline in this final report are suggestive of potential measures that could cut across the four homeless-serving programs. It is recommended that careful thought be given to the development and implementation of such measures so that programs are not burdened by large numbers of overly complicated performance measures. Each measure added will likely require program staff to make changes in data collection forms, procedures, and automated data systems, as well as likely impose added burden and costs on program staff and participants. However, given the increasing emphasis on measurement of program performance in recent years by Congress and the potential for performance data to provide valuable feedback for enhancing service delivery, it is critical to identify potential ways in which programs can better track participant outcomes  particularly, changes in status (e.g., housing situation or earnings) from the time of entry into homeless-serving programs through termination and beyond.

Building on outcome measures suggested in this report and moving beyond the specific programmatic outcomes for participants in the four DHHS homeless-serving programs that are the focus of this study, it may be possible down the road to introduce (1) experimental designs for measuring net impacts of program services and (2) system-wide measures that communities may be able to use to gauge the overall success of their efforts to counter problems associated with homelessness. Such experimental designs could employ some of these same outcome measures, but compare outcomes (e.g., whether days of homelessness are reduced or whether labor force attachment and earnings increase) for individuals receiving program services versus similar outcomes for a randomly assigned control group of homeless individuals (not receiving services). Introduction of system-wide measures could provide the opportunity for exploring the wider potential effects of a group of or all homeless services within a particular locality (as well as other contextual factors, such as local economic conditions and loss of affordable housing). Such system-wide measures would not be used to hold individual programs accountable for achievement of specified outcomes, but rather enable state and local decision-makers (e.g., a mayor of a large metropolitan area) to address more expansively questions about the local homeless situation, such as is the problem of chronic homelessness intensifying in the community or is the community making a dent in the number of homeless individuals on the streets and living in emergency shelters each night or to what extent is the community addressing its general homeless problem.

Endnotes

(15) HCH is clustered with several other programs, including Community Health Centers [CHCs], Migrant Health Centers, Health Services for Residents of Public Housing, and other community-based health programs.

(16) At the same time, each program has more specific goals which relate to the populations served and related to its original program intent  for example, RHYs BCP component has as one of its goals family reunification (when appropriate); HCH aims to improve health care status of homeless individuals; PATH aims to engage participants in mental health care services and improve mental health status; and the Treatment for Homeless Persons program aims at engaging participants in substance abuse treatment and reducing/eliminating substance abuse dependency.

Application of Suggested Core Performance Measures to DHHS Mainstream Programs Serving Homeless Individuals

While the main focus of this study is on examining the extent to which the four selected homeless-serving programs could potentially enhance performance measurement through adoption of a core set of performance measures, this study is also intended to assess the potential applicability of the suggested core measures to mainstream DHHS programs that serve both homeless and non-homeless populations. Of critical interest is assessing the capability and willingness of mainstream DHHS programs to (1) collect basic data relating to the number and types of homeless individual served, and (2) move beyond counts of homeless individuals served to adopting some or all the core performance measures suggested in Chapter 4.

With input from the DHHS Project Officer, we selected four DHHS mainstream programs for analysis: (1) the Health Centers Cluster (administered by Health Resources and Services Administration [HRSA]), (2) the Substance Abuse Prevention and Treatment (SAPT) Block Grant (administered by Substance Abuse and Mental Health Services Administration [SAMHSA]), (3) Head Start (administered by Administration for Children and Families [ACF]), and (4) Medicaid (administered by the Centers for Medicaid and Medicare Services [CMS]). While these programs are not targeted specifically on homeless individuals, some homeless individuals are eligible for services provided under each program by virtue of low income, a disability, or other characteristics. In fact, each of these mainstream programs serves some homeless individuals, though homeless individuals constitute a relatively small share of total individuals served within each program. We conducted telephone interviews with officials at each of the agencies administering these programs to collect information about each of the programs, as well as views on collecting additional data on services to homeless individuals. In addition, we reviewed information about the data systems supporting collection of performance data and reporting requirements (including GPRA performance measures) for each program.

Main Findings from Interviews with Administrators and Reviews of Background Documentation on DHHS Mainstream Programs

In our interviews with program administrators and reviews of background documents, we first sought to develop a basic description of each of the four mainstream programs, then focused on the following: (1) data systems used to collect performance data and types of process and outcome measures regularly collected on program participants, (2) whether programs tracked homeless individuals and, if so, estimates of the number of homeless served, (3) specific data elements collected on homeless individuals served by the program (if any), (4) GPRA measures currently in use, (5) agency views on their ability to track numbers/percentage of homeless individuals served (if not already tracking this), (6) agency views about feasibility of collecting data relating to the suggested core performance measures (i.e., the measures shown earlier in Exhibit 4-1), and (7) agency views on difficulties involved in making changes to existing data systems that would be necessary for enhanced tracking of homeless individuals served and their outcomes.

Overview of the Four Mainstream Programs

Basic characteristics of mainstream program generally quite different in terms of scale and target population. As shown in Exhibit 5-1, the four mainstream programs are quite different on a number of important dimensions from the four homeless-serving programs that are the main focus of this study. In comparison to the four homeless-serving programs, the mainstream programs:

• Have much greater funding  The largest of the four homeless-serving programs in terms of budget is the Health Care for the Homeless (HCH) program, with an annual budget of slightly more than $100 million. The funding levels of HCH and the other homeless-serving programs pale in comparison to those of the four mainstream programs: Medicaid, with FY 2002 federal assistance to states of $147.3 billion; Head Start, with a FY 2002 budget of $6.5 billion; SAPT, with a FY 2002 budget of $1.7 billion, and the BPHCs Health Centers Cluster, with FY 2002 budget of $1.3 billion (which includes funding for HCH).
• Serve many more individuals  As might be expected given their greater funding levels and mandates to serve a broader range of disadvantaged individuals, the mainstream programs enroll and serve many more individuals in 2002, Medicaid had nearly 40 million enrolled beneficiaries, far eclipsing the other mainstream and homeless-serving programs. In 2001, the Health Centers Cluster served an estimated 10.3 million individuals, while SAPT served an estimated 1.6 million individuals (in FY 2000) and Head Start enrolled nearly a million (912,345 in FY 2002) children.
• Serve a generally more broadly defined target population  While similarly targeted on low-income and needy individuals, the mainstream programs extend program services well beyond homeless individuals. Of the four mainstream programs, the two broadest programs are the Medicaid and Health Cluster Centers programs, both focusing on delivery of health care services to low-income and disadvantaged individuals. For example, though there is considerable variation from state to state, individuals may qualify for Medicaid benefits as part of either mandatory or categorically need groups.⁽¹⁷⁾ The Head Start program targets needy and low-income pre-schoolers ages 3 to 5 (90 percent of which must meet low-income guidelines). The program also extends a range of services to the parents of these children to assist them in being better parents and educators of their children. SAPT is primarily targeted on individuals who abuse alcohol and other drugs, but also extends preventive educational and counseling activities to a wider population of at-risk individuals (i.e., not less that 20 percent of block grant funds are to be spent to educate and counsel individuals who do not require treatment and provide activities to reduce risk of abuse).

Despite some differences, there are commonalities in terms of program goals and services offered by mainstream and homeless-serving programs. Three of the four mainstream programs (Medicaid, SAPT, the Health Centers Cluster) focus program services primarily on improving health care status of low-income individuals through provision of treatment and preventative care. Two of the programs  Medicaid and the Health Centers Cluster  are aimed directly at delivery of health care services to improve health care status of low-income and needy individuals. Though more narrowly targeted on homeless individuals, HCH and PATH are similarly aimed at improving health care status of the disadvantaged individuals. The third mainstream program  SAPT aims at improving substance abuse treatment and prevention services. Under SAPT, block grants funds are distributed to states, territories, and tribes aimed at the development and implementation of prevention, treatment, and rehabilitation activities directed to diseases of alcohol and drug abuse. Program services sponsored under SAPT (i.e., the treatment services) are perhaps most similar to the Treatment for Homeless Persons and PATH programs (though PATH has additional focus on provision of mental health services).

In terms of program goals and services, the fourth mainstream program  Head Start  is quite different from the three other mainstream programs and the four homeless-serving programs. The Head Start program is aimed principally at increasing school readiness and social competence of young children in low-income families. The program promotes school readiness by enhancing the social and cognitive development of children through the provision educational, health, nutritional, social, and other services. Head Start also engages parents in their children's learning and assists parents in making progress toward their educational, literacy, and employment goals.⁽¹⁸⁾

Types of Performance Data Collected and Prospects for Including Additional Data About Homeless Individuals in Mainstream Programs

All four of the mainstream programs have well-established data systems which are used to collect data on characteristics of those served, services received, and results of service delivery. These data  along with other special surveys and data sources  are used by the federal agencies overseeing each of these programs for GPRA reporting and to generally monitor program performance. As is discussed below, the four mainstream programs collect minimal (if any) data on the homeless status of program participants and, for the most part, do not place a high priority on collecting additional data concerning the homeless individuals they serve (especially in light of budgetary constraints and many competing demands that agencies face for generating performance data on their programs).

Two of the four mainstream programs  Head Start and the Health Centers Cluster  track numbers of homeless served; Medicaid and SAPT do not track number of homeless served. As shown in Exhibit 5-2, Head Start grantees are required to submit an annual report (known as the Program Information Report, PIR) to the Head Start Bureau that includes data on enrollment levels, child/family characteristics, center staffing and program services, and participant outcomes.⁽¹⁹⁾ As part of the PIR, each center reports the total numbers of children and families served, as well as aggregate number of homeless children and families served during the enrollment year.⁽²⁰⁾ The other data item submitted (as part of the PIR) pertaining to homelessness is the number of homeless families that acquired housing during the enrollment year.

The other mainstream program that collects data on the number of homeless individuals served is the Health Centers Cluster. As part of the Uniform Data System (UDS), all grantees (primarily CHCs) are required to report annually on the number of users known to be homeless at some time during the reporting period. Grantees submit aggregate counts and are permitted to submit estimates of homeless users. In addition, Health Center Cluster grantees with HCH sites (one of the Center Cluster programs) are also required to provide separate annual counts of homeless users in (1) homeless shelters, (2) transitional, (3) doubling up, (4) street, (5) other, and (6) unknown living situation.

There are no separate breakouts or counts of homeless individuals served submitted to the federal government as part of the Medicaid or the SAPT block grant programs data systems. Under Medicaid, states may elect to include homelessness or living situation in their state Medicaid data systems. However, the federal government does not track which states collect data on number of homeless individuals served and the MMIS (the reporting system by which Medicaid providers report on services to Medicaid beneficiaries to the federal government) does not include data elements that would enable providers to report on the number of homeless served.⁽²¹⁾

SAPT grantees use the Treatment Episode Data Set (TEDS) to collect client-level data, including the following core data elements: client identifiers, client characteristics (date of birth, sex, race, etc.), date of admission, types of services client received, source of referral, employment status, substance abuse problem, and frequency of use). While the required portion of the TEDS does not include a data item to track homelessness, CSAT has made available a TEDS supplemental data set (which grantees can elect to use) that does include a variable designed to capture living arrangement at the time of intake. Grantees using the supplemental data set, are provided with three living arrangement choices to capture at the time of enrollment: (1) homeless (clients with no fixed address; includes shelters); (2) dependent living (clients living in supervised setting such as a residential institution, halfway house or group home; and (3) independent living (clients living alone or with others without supervision).

Estimates of the number of homeless served are available for one of the four mainstream programs. In our interviews and review of data, we could obtain a firm estimate of the number of homeless served only from the Head Start program  estimated at about 2 percent of all those children enrolled in Head Start. The data system utilized by BPHCs Health Centers Cluster grantees  the Uniform Data System  includes a field for grantees to report either actual or estimates of the number of homeless individuals served. Although this data field should enable users to estimate roughly the total percentage of users of Health Center Cluster services that are homeless, we were unable to obtain such a current estimate. Estimates of the percentage of homeless served are not available for the Medicaid or SAPT program (i.e., the federal government does not collect this data from states, though such estimates may be available for individual states.⁽²²⁾

Three of the four mainstream programs (all except Medicaid) provide guidance on the definition of homeless. Explicit guidance on how to define homeless individuals served is provided as part of the automated reporting systems for the Head Start and Health Centers Clusters programs. The annual Performance Information Report (PIR) submitted by Head Start grantees to the federal government provides the following guidance on how to define homelessness: Homeless families include those that live temporarily in shelters, motels, or vehicles and families that move frequently between the homes of relatives or friends. Include all families that had any period of homelessness during the enrollment year. Similarly, the Uniform Data System (UDS) report (submitted annually by each Health Center Cluster grantees) provides guidance on what constitutes a homeless individual: individuals who lack housing, including individuals whose primary residence during the night is a supervised public or private facility that provides temporary living accommodations, and individuals who reside in transitional housing. The TEDS supplemental data set  which may be used by SAPT block grant providers, but is not required  includes living arrangement as a variable that may be collected at the time of intake to SAPT services. As noted earlier, three living arrangement choices are provided (homeless, dependent living, and independent living) and homeless is defined as clients with no fixed address; includes shelters.

With the possible exception of counts of homeless individuals served, the mainstream programs do not collect sufficient information to address the suggested core performance measures (identified in Chapter 4, Exhibit 4-1). Data are being currently collected at the federal level by only two of the mainstream programs  Head Start and the Health Centers Cluster  on any of the suggested core performance measures identified in Chapter 4. Both of these programs can generate an overall count of homeless individuals served that addresses the first suggested performance measure (i.e., number of homeless individuals enrolled). The Health Centers Cluster also collects a second measure as part of the annual Performance Information Report submitted by grantees  the number of homeless families who acquired housing during the enrollment year  that is somewhat similar to another suggested core outcome measures: number/percent of homeless individuals enrolled whose housing condition is upgraded during the past month [or quarter]. The other two mainstream programs  Medicaid and SAPT  do not collect data on homeless status at the federal level  and hence, it is not possible to generate the data needed to address any of the suggested core measures.

Mainstream program GPRA measures are combination of process- and outcome-oriented measures and are not closely aligned with suggested core performance measures for homeless-serving programs. Exhibit 5-3 displays the GPRA measures for the four mainstream programs, along with several of the sources used to collect client level data and report on GPRA measures. Appendix D provides additional details about the GPRA measures and specific targets for key measures. Each of the programs has a set of measures that are tailored to specific goals of the program. All four of the programs include both process and outcome measures, though the Head Start program generally places more emphasis on outcomes for individuals served, while the other three mainstream programs tend to place somewhat more emphasis on process measures. For example, Head Starts includes GPRA measures and targets for improved cognitive skills, improved gross and fine motor skills, improved emergent literacy, numeracy, and language skills, etc.⁽²³⁾ Perhaps of greatest relevance to this study  none of the mainstream programs have GPRA measures that refer to serving homeless individuals or to improving outcomes for homeless individuals. Overall, as they are currently structured, the GPRA measures for the four mainstream programs are not aligned with the suggested performance measures for the homeless-serving programs (shown earlier in Chapter 4).

Exhibit 5-3:
Overview of FY 2004 GPRA Measures and Data Sources
Used to Collect Participant and Performance Measurement Data

Program	Overview of GPRA Measures	Methods Used to Collect Participant and Performance Data
Head Start	FY 2004 GRPR performance goals/measures organized as follows: Goal #1: Enhance childrens growth and development  Children demonstrate improved emergent literacy, numeracy and language skills. Children demonstrate improved general cognitive skills. Children demonstrate improved gross and fine motor skills Children demonstrate improved positive attitudes toward learning. Children demonstrate improved social behavior and emotional well-being. Children demonstrate improved physical health Head Start parents demonstrate improved parenting skills Parents improve their self-concept and emotional well-being. Parents make progress toward their education, literacy and employment goals Goal #2: Children receive educational services  Programs provide developmentally appropriate educational environments Staff interact with children in a skilled and sensitive manner Goal #3: Children in Head Start receive health and nutritional services  Children in Head Start receive needed medical, dental and mental health services	All Head Start programs submit annual Program Information Report (PIR). PIR includes data about the center, enrollment levels, participant and family characteristics, and limited outcome data on the children and families. PIR data are at aggregate level by center. Family and Child Experiences Survey (FACES) is periodic (in 3-year cycles), longitudinal survey of nationally representative sample of Head Start children and families. FY 1999 baseline FACES survey conducted with 3,200 children and families in 40 programs. Survey gathers individual-level data on cognitive, social, emotional and physical development of Head Start children, as well as well-being of families and quality/ characteristics of Head Start classrooms. PIR, FACES, and other sources used to generate data for GPRA reporting.
Medicaid	CMS FY 2004 GPRA performance goals/measures organized as follows: Increase the percentage of Medicaid 2-year old children who are fully immunized (supports DHHS Strategic Goal #1: Reduce the major threats to the health and well-being of Americans and Goal #7: Improve the stability and healthy development of our Nations children and youth) Assist states in conducting Medicaid payment accuracy studies for the purpose of measuring and ultimately reducing Medicaid payment error rates (supports DHHS Strategic Goal #8: Achieve excellence in management practices) Improve health care quality across Medicaid and the State Childrens Health Insurance Program (SCHIP) (supports DHHS Strategic Goal #5: Improve the quality of health care services) Decrease the number of uninsured children by working with states to implement SCHIP and by enrolling children in Medicaid (supports DHHS Strategic Goal #3: Increase the percentage of the Nations children and adults who have access to regular health care services and expand consumer choices)	Medicaid Management Information System (MMIS) is main data system states are required to have for collection of beneficiary data. Variety of data sources used for reporting on GPRA measures. For example, Health Plan Employer Data Information Set (HEDIS), the Clinical Assessment and Software Application (CASA), and immunization registries provide standardized measurement of childhood immunization (for the first goal).
SAPT	FY 2004 GPRA performance goals/measures organized as follows: Number of clients served (annual targets set) Increase # states/territories voluntarily reporting measures in SAPT Block Grant application Increase % states that express satisfaction with TA provided Increase % of TA events that result in systems, program, or practice changes Increase % of Block Grant applications that include needs assessment data Increase % of states that indicate satisfaction with CSAT customer services throughout the entire Block Grant process Increase % of states reporting satisfaction with CSATs responsiveness to state suggestions on services.	SAPT grantees use Treatment Episode Data Set (TEDS) to collect client- level data, including following core data elements: client identifiers, client characteristics (DOB, sex, race, etc.), date of admission, types of services client received, source of referral, employment status, substance abuse problem, and frequency of use. TEDS supplemental data elements include: living arrangement, DSM criteria, sources of income, and health insurance TEDS Discharge Data Set collects types of services at discharge, date of discharge, date of last contact, and reasons for discharge
Health Centers Cluster	FY 2004 GPRA performance goals/measures are organized as follows: Goal #1: Eliminate barriers to care: Increase utilization  (1) By 2006, establish additional 1,200 new or expanded sites; serve 6.1 million more patients; (2) Increase # of uninsured and underserved persons serve by Health Centers, with emphasis on areas with high % of uninsured children Increase access point  (1) Increase infrastructure of Health Center program to support an increase in utilization, via new starts, new satellite sites, & expanded sites; (2) Among new grant applicants, increase # of faith-based or CBOs Focus on target pop.  (1) By 2006, assure access to most vulnerable by serving 16% of nations pop. at or below 200% of poverty; (2) Continue to assure access to preventive and primary care for (a) minority and (b) uninsured individuals Goal #2: Eliminate health disparities: Utilization of services  (1) Increase % of diabetic users with up-to-date testing of glycohemoglobin and who have annual dilated eye exam; (2) By 2006, reduce low birth weight rates to 6.53%; (3) Decrease % of births to prenatal care Health Center users to 2500 grams (LBW); (4) Increase % of Health Center women receiving age-appropriate screening for cervical and breast cancer, (5) Increase % of Health Center adults with hypertension who report blood pressure under control Goal#3: Assure Quality of Care: Decrease % of Health Center users hospitalized for potentially avoidable conditions	Each grantee submits standardized set of tables annually using Uniform Data System (UDS) Grantees submit 9 UDS tables reporting aggregate (not individual-level) data, such as: users of services by age category, race/ethnicity, income category, type of 3rd party insurance source, staffing levels at facilities by type of personnel, and numbers of encounters Grantees aggregate data on most UDS tables for all Cluster programs, except Tables 3 and 4 (demographics), and Table 6 (users and encounters by diagnostic category) BPHC also conducts User Visits Survey of a representative sample of health center users and provider visits  which provides data on individuals served and the care they receive.

Mainstream programs face substantial constraints to making changes to existing data systems to increase tracking of homeless individuals. Our discussions with program administrators suggest that the prospects for going much beyond tracking whether an individual is homeless are not promising  and implementation of additional performance measures such as those suggested in Chapter 4 is likely to be a non-starter with mainstream programs. There are two main hurdles that would need to be overcome to expand tracking of homeless individuals by mainstream programs. First, as large mainstream programs, the homeless typically represent a relatively small proportion of total individuals served  and are not typically a population of primary interest. For example, as noted earlier, in a program such as Head Start, homeless account for no more than about two percent of the total number children enrolled at centers. Capturing additional data on homeless individuals serves competes poorly with other critical information needs faced by these mainstream programs and hence, are likely to be viewed well down on the list of must have data elements that such programs are seeking to obtain. Second, adding new data items to existing systems  especially programs such as Medicaid and Head Start which have with well-established data system  is a costly proposition for federal and state agencies, as well as burdensome for service providers and participants served by these programs. The federal government will likely need to negotiate with state agencies and/or grantees on planned changes to such data systems and may encounter resistance or requests for additional funding even when agreement is reached over the addition of new data or reporting elements. For example, a Medicaid program official we interviewed underscored the difficulties involved  noting that at both the federal and state level, it would be a huge problem to mandate tracking of homeless individuals served. This would mean a redesign of the MMIS, which would be very time consuming and expensive. Given the current budgetary environment (with states/localities facing financial difficulties/crises), this Medicaid official observed that it would be very difficult to impose new reporting requirements on states. A Head Start official noted that the program had conducted a major redesign of its grantee performance reporting system just two years ago (adding three new homeless measures, among many other changes). He noted that this recent revision required considerable time and effort at all levels  and that making changes to the data system used by Head Start so soon after it had been extensively restructured would be a difficult row to hoe.

Implications and Conclusions

Despite constraints, mainstream programs should be encouraged to collect data on living arrangement (or homeless status) at time of enrollment, and periodically, to collect more in-depth information about homeless individuals served as part of special surveys or studies. Recognizing the difficulties faced by the mainstream programs in making changes to their well-established data sets, it would be very useful to work with mainstream DHHS programs to: (1) add a single data element to data systems that would capture living arrangement or homeless status at the time of program enrollment in a consistent manner across programs; (2) provide the mainstream programs with a common definition of what constitutes homelessness and, if possible, the specific question(s) and close-ended response categories that programs should use in tracking homelessness; and (3) if mainstream programs conduct a follow-up interview or survey with participants, request that they include a follow-up question relating to homelessness or living arrangement. Two of the four mainstream programs  Head Start and the Health Centers Cluster  already track homeless individuals served as part of their current grantee reporting systems. These two programs demonstrate the feasibility of mainstream programs tracking homeless individuals served. In the case of these two programs, it would be beneficial if a common definition of homelessness was used and if the same question(s) and response categories were asked of program participants at the time of intake.

The other two mainstream programs  Medicaid and SAPT  extend considerable flexibility to grantees to track homelessness or living arrangement if they desire to do so, but do not require these data to be submitted the federal government. With respect to SAPT, it makes sense to negotiate with states to expand use of the TEDS supplemental data set, which provides an indicator of living situation at the time of enrollment. In addition, as noted above, to the extent possible, it would be desirable to use a common definition of homelessness, as well as common question(s) and response categories.

Medicaid is a special case. The sheer size of the Medicaid program and high costs associated with adding new data elements to the MMIS, may simply not make it possible to track homelessness at the time of enrollment as part of the MMIS. However, addition of an indicator of homeless status at the time of enrollment would be very useful  especially given the very large number of beneficiaries of the program. As a first step, inquiries should be made to states to determine which states/localities may already be tracking homelessness as part of their intake forms or state data systems. Such an inquiry would be helpful both from the standpoint of determining the extent to which states/localities are already collecting such data, as well as determining potential advantages and drawbacks of collecting data relating to homelessness or living arrangement. Further discussions are also needed with Medicaid program officials to determine when the next round of changes to the MMIS is expected, as well as the steps required to include tracking of homeless individuals served. One of the critical advantages of adding a variable that would identify an individuals living arrangement at the time of intake is that given that the system collects individual (beneficiary) data, it would be possible to not only generate an overall count of the numbers of homeless Medicaid beneficiaries, but also to conduct more in-depth analyses of characteristics, services received, and costs of services for participants according to living arrangement at the time of entry into the program.

For all four of the mainstream programs and the four homeless-serving programs, a step beyond collecting homeless status or living arrangement at the time of enrollment would be to collect such data at the time of exit from the program or at some follow-up point following enrollment or termination from the program (e.g., 6 months, 12 months, or later). However, determining a convenient follow-up point to interact with the participant may be difficult or impossible in these programs. With regard to collecting homeless or living arrangement status at a follow-up point, it may be best to focus (at least first) on implementing such follow-up measures in the homeless-serving programs, where long-term housing stability is a critical program objective.

Finally, where collection of information about homeless status either at the time of enrollment or some follow-up point prove either impossible to obtain or too costly, DHHS should consider potential opportunities for collecting data on homelessness as part of special studies or surveys. Several of the mainstream programs (as well as the homeless-serving programs) are periodically the subject of either special studies or survey efforts. For example, the Head Start program has implemented the FACES survey, which is conducted in 3-year waves on a sample of over 3,000 children and families served by 40 Head Start centers. Working with a sample, rather than in the universe in large programs such as Head Start (nearly 1 million children) and Medicaid (about 40 million beneficiaries) has great appeal from the standpoint of reducing burden and data collection costs. In addition, such smaller survey efforts may present an opportunity for including many more specialized questions (e.g., concerning homelessness) and tracking change in housing situation over time (i.e., pre/post comparisons).

Endnotes

(17) Medicaid mandatory groups include: low-income families with children meeting TANF eligibility, Supplemental Security Income (SSI) recipients, infants born to Medicaid-eligible women, low-income children (under age 6) and pregnant women, recipients of adoption assistance and foster care, and certain low-income Medicare beneficiaries. Categorically needy groups include: income pregnant women, certain aged, blind or disabled adults, low-income children under age 21 that are not eligible for TANF, low-income institutionalized individuals, persons who would be eligible if institutionalized but are receiving care under community-based services waivers, recipients of state supplementary payments, and low-income, uninsured women screened and diagnosed and determined to be in need of treatment for breast or cervical cancer.

(18) RHY program similarly focuses on youth  though Head Start focuses services on a much younger age cohort (3-5) and is much more focused on developmental activities and getting very young children onto the right path.

(19) The Head Start Bureau initiated a redesign of the PIR in 2001 (the PIR Redesign Project) that led to the major revamping of the PIR report for the 2002 enrollment year. As part of this redesign, three measures related to homelessness were added to the PIR.

(20) Head Start centers submit aggregate data to the federal government as part of the PIR (i.e., participant-level data are not submitted).

(21) Because homelessness is not a condition that relates to being eligible for Medicaid benefits, it is not a data element reported by states through the MMIS to the federal government.

(22) Given the very large number of beneficiaries of the Medicaid program (about 40 million), homeless individuals likely make up a relatively small proportion of total Medicaid beneficiaries.

(23) Though not shown on the exhibit, Head Start has specific targets for each goal  e.g., under the objective of children demonstrate improved emergent literacy, numeracy and language skills among the targets set are achieve at least an average 34 percent gain (12 scale points) in word knowledge for children completing the Head Start program.

Appendices

Discussion Guide for Interviews with Administrators and Staff of Four Homeless-Serving Programs

Discussion Guide for Initial Visits to
DHHS Homeless-Serving Programs:
Addictions Treatment for Homeless Persons Program

Interviewee:
Date of Interview:

Background on the Program and Services 

1. .Please provide a brief history of the programs origins and evolution.
2. What is the most recent annual budget for the program? What are the budget requests for the next several out years? Is all the money allocated via SGAs, such as the one on the web, or is some of the money also distributed by formula? If other distribution means are used, please describe.
3. Please describe the general structure of the program:
   1. What is the role of the federal government?
   2. Is there a specific role for states, or is their role limited to their role as grantees?
   3. Who are the grantees? What is the distribution among states, cities and counties, CBOs, and other organizations?
   4. How common is it for a grantee to provide all or most of the programs services as opposed to using subcontractors/sub-grantees? What is the nature and distribution of the subcontractors/sub-grantees?
4. The agencys GPRA plan and outcome measures are attached to the SGA. Please answer the following questions:
   1. Are all the GPRA measures of equal importance to the agency? If not, please rank them.
   2. Are there any other performance measures beyond the GPRA measures that you currently use? If so, what are they, how are they tracked, and how do they rank compared to the GPRA measures?
5. The SGA indicates that grantees are required to conduct local evaluations. Would you characterize these evaluations as process studies, outcome studies, or net impact evaluations? How do you use the local evaluations in measuring performance of your grantees? How do the local evaluations tie in to your GPRA work? Can you provide us with several examples of local evaluations, including some that are good and some that are poor in quality?
6. Please provide annual enrollment and budget data for the past three years for the program and current year budget and target for enrollment.
7. On page 6 of the SGA, the target population for the SGA is described. Does this apply to all uses of the programs funds? Are some subgroups considered of higher priority than others?
8. How competitive is the program? What proportion of grant applications are funded? What differentiates those funded from those not funded?
9. Once recruited, how do participants flow through the program (i.e., from recruitment, through intake and assessment, and into program services)?
10. Please briefly describe the main types of services and activities participants receive (i.e., health care services; mental health services; substance abuse services; employment and training services; help resolving housing problems; other support services)  [note: indicate whether all participants receive a specific service or only some participants receive the service]?
11. Do local programs typically partner with other agencies to provide services to participants (i.e., refer participants to other agencies for services)? If so, who are the other organizations and what services do they provide? How does the partnering work? Is there a subcontract, formal referral, informal referral, or some other approach?
12. At what point in the process is an individual considered a program participant? Are program participants formally terminated from the program, and if so, when? How long do participants typically stay enrolled in the program (i.e., range/average length of involvement)? Is there a problem with attrition (before participants complete program services)?
13. What do you believe to be the major impacts/effects of the program on participants? (Indicate the outcomes and the levels typically achieved)

Performance Measurement and Information Flow:

1. Please provide us with copies of all data collection requirements and forms for grantees. If you have reports from previous years, please provide us with copies. What data do you require beyond the GPRA outcomes, and why do you require it?
2. Explain all the uses you make of data reported by grantees, including rewards, sanctions, use in future funding, completing GPRA reports, etc.
3. Do you tie the local evaluation data in any way to the GPRA data in assessing the program?
4. Has the federal office implemented a standardized automated data system across program grantees to collect performance data? [If yes, obtain documentation on the data system, such as a copy of participant forms.] Alternatively, have grantees developed their own automated data systems? If yes, please provide a brief description of the various types of systems in use (e.g., is there great diversity in the types of systems in use)?
5. How satisfied are program administrators with the existing performance monitoring and/or data system? Are there ways in which the performance monitoring or data systems might be improved? Is there any planned or ongoing effort to change either the types of performance information collected or the way in which these data are collected?
6. What types of information would your agency ideally want to be able to measure regarding program performance (e.g., process and outcome measures)? To what extent would it be possible for states/localities to report on these measures?
7. As part of this study, we are exploring possible methodologies for identifying chronically homeless individuals. Our aim will be to develop a brief set of questions, characteristics, proxy measures, or indices that could be readily and efficiently determined at intake/enrollment in a program and which would assist in identifying a chronically homeless client. Do you have any suggestions of possible questions/characteristics that should be included in this index?
8. We were a bit confused by Appendix B on the agencys GPRA plan, Could you please walk us through it and explain how you are implementing it for your program?
9. Appendix C provides an OMB approved client survey instrument. Could you provide us a description of how the data are collected and used?
10. Because this program will usually be one of several serving the participants, either concurrently or sequentially, do you think the performance measures used or proposed can isolate the effects of this program? Do you have any suggestions for dealing with this?

Discussion Guide for Initial Visits to
DHHS Homeless-Serving Programs:
Health Care for the Homeless

Interviewee:
Date of Interview:

Background on the Program and Services 

1. Please provide a brief history of the programs origins and evolution.
2. What is the most recent annual budget for the program? What are the budget requests for the next several out years? How are funds allocated/distributed to states/local grantees (e.g., formula, competitive process)? If other distribution means are used, please describe.
3. Please describe the general structure of the program:
   1. What is the role of the federal government?
   2. What is the role for states?
   3. Who are the grantees? What is the distribution among states, cities and counties, CBOs, and other organizations?
   4. How common is it for a grantee to provide all or most of the programs services as opposed to using subcontractors/sub-grantees? What is the nature and distribution of the subcontractors/sub-grantees?
4. Please provide a copy of the GPRA plan and outcome measures for this program and address the following questions:
   1. Are all the GPRA measures of equal importance to the agency? If not, please rank them.
   2. Are there any other performance measures beyond the GPRA measures that you currently use? If so, what are they, how are they tracked, and how do they rank compared to the GPRA measures?
5. Are there any national, state, or local evaluations of the program? If so, would you characterize these evaluations as process studies, outcome studies, or net impact evaluations? How do you use the evaluations in measuring performance of your grantees? How do the evaluations tie in to your GPRA work? Can you provide us with several examples of evaluations, including some that are good and some that are poor in quality?
6. If any HCH funding is distributed to states, how do the states distribute their funds? Do they sometimes run programs themselves?
7. How competitive is the program? What proportion of grant applications are funded? What differentiates those funded from those not funded?
8. Once recruited, how do participants flow through the program (i.e., from recruitment, through intake and assessment, and into program services)?
9. Please briefly describe the main types of services and activities participants receive (i.e., health care services; mental health services; substance abuse services; help resolving housing problems; other support services)  [note: indicate whether all participants receive a specific service or only some participants receive the service]?
10. Do local programs typically partner with other agencies to provide services to participants (i.e., refer participants to other agencies for services)? If so, who are the other organizations and what services do they provide? How does the partnering work? Is there a subcontract, formal referral, informal referral, or some other approach?
11. At what point in the process is an individual considered a program participant? Are program participants formally terminated from the program, and if so, when? How long do participants typically stay enrolled in the program (i.e., range/average length of involvement)? Is there a problem with attrition (before participants complete program services)?
12. What do you believe to be the major impacts/effects of the program on participants? (Indicate the outcomes and the levels typically achieved)

Performance Measurement and Information Flow:

1. Please provide us with copies of all data collection requirements and forms for grantees. If you have reports from previous years, please provide us with copies. What data do you require beyond the GPRA outcomes, and why do you require it?
2. Explain all the uses you make of data reported by grantees, including rewards, sanctions, use in future funding, completing GPRA reports, etc.
3. Do you tie the evaluation data in any way to the GPRA data in assessing the program?
4. Has the federal office implemented a standardized automated data system across states and program grantees to collect performance data? [If yes, obtain documentation on the data system, such as a copy of participant forms.] Alternatively, have grantees developed their own automated data systems? If yes, please provide a brief description of the various types of systems in use (e.g., is there great diversity in the types of systems in use)?
5. How satisfied are program administrators with the existing performance monitoring and/or data system? Are there ways in which the performance monitoring or data systems might be improved? Is there any planned or ongoing effort to change either the types of performance information collected or the way in which these data are collected?
6. What types of information would your agency ideally want to be able to measure regarding program performance (e.g., process and outcome measures)? To what extent would it be possible for states/grantees to report on these measures?
7. As part of this study, we are exploring possible methodologies for identifying chronically homeless individuals. Our aim will be to develop a brief set of questions, characteristics, proxy measures, or indices that could be readily and efficiently determined at intake/enrollment in a program and which would assist in identifying a chronically homeless client. Do you have any suggestions of possible questions/characteristics that should be included in this index?
8. Because this program may be one of several serving the participants, either concurrently or sequentially, do you think the performance measures used or proposed can isolate the effects of this program? Do you have any suggestions for dealing with this?

Discussion Guide for Initial Visits to
DHHS Homeless-Serving Programs:
PATH

Interviewee:
Date of Interview:

Background on the Program and Services 

1. Please provide a brief history of the programs origins and evolution.
2. What is the most recent annual budget for the program? What are the budget requests for the next several out years? Is all the money allocated by formula? If other distribution means are used, please describe.
3. Please describe the general structure of the program:
   1. What is the role of the federal government?
   2. What is the role for states?
   3. Who are the grantees? What is the distribution among states, cities and counties, CBOs, and other organizations?
   4. How common is it for a grantee to provide all or most of the programs services as opposed to using subcontractors/sub-grantees? What is the nature and distribution of the subcontractors/sub-grantees?
4. Please provide a copy of the GPRA plan and outcome measures for this program and address the following questions:
   1. Are all the GPRA measures of equal importance to the agency? If not, please rank them.
   2. Are there any other performance measures beyond the GPRA measures that you currently use? If so, what are they, how are they tracked, and how do they rank compared to the GPRA measures?
5. Are there any national, state, or local evaluations of the program? If so, would you characterize these evaluations as process studies, outcome studies, or net impact evaluations? How do you use the evaluations in measuring performance of your grantees? How do the evaluations tie in to your GPRA work? Can you provide us with several examples of evaluations, including some that are good and some that are poor in quality?
6. How do the states distribute their funds? Do they sometimes run programs themselves? Do they sponsor competitions and/or do they distribute the funds to county and local government?
7. How competitive is the program? What proportion of grant applications are funded? What differentiates those funded from those not funded?
8. Once recruited, how do participants flow through the program (i.e., from recruitment, through intake and assessment, and into program services)?
9. Please briefly describe the main types of services and activities participants receive (i.e., health care services; mental health services; substance abuse services; employment and training services; help resolving housing problems; other support services)  [note: indicate whether all participants receive a specific service or only some participants receive the service]?
10. Do local programs typically partner with other agencies to provide services to participants (i.e., refer participants to other agencies for services)? If so, who are the other organizations and what services do they provide? How does the partnering work? Is there a subcontract, formal referral, informal referral, or some other approach?
11. At what point in the process is an individual considered a program participant? Are program participants formally terminated from the program, and if so, when? How long do participants typically stay enrolled in the program (i.e., range/average length of involvement)? Is there a problem with attrition (before participants complete program services)?
12. What do you believe to be the major impacts/effects of the program on participants? (Indicate the outcomes and the levels typically achieved)

Performance Measurement and Information Flow:

1. Please provide us with copies of all data collection requirements and forms for grantees. If you have reports from previous years, please provide us with copies. What data do you require beyond the GPRA outcomes, and why do you require it?
2. Explain all the uses you make of data reported by grantees, including rewards, sanctions, use in future funding, completing GPRA reports, etc.
3. Do you tie the evaluation data in any way to the GPRA data in assessing the program?
4. Has the federal office implemented a standardized automated data system across states and program grantees to collect performance data? [If yes, obtain documentation on the data system, such as a copy of participant forms.] Alternatively, have grantees developed their own automated data systems? If yes, please provide a brief description of the various types of systems in use (e.g., is there great diversity in the types of systems in use)?
5. How satisfied are program administrators with the existing performance monitoring and/or data system? Are there ways in which the performance monitoring or data systems might be improved? Is there any planned or ongoing effort to change either the types of performance information collected or the way in which these data are collected?
6. What types of information would your agency ideally want to be able to measure regarding program performance (e.g., process and outcome measures)? To what extent would it be possible for states/grantees to report on these measures?
7. As part of this study, we are exploring possible methodologies for identifying chronically homeless individuals. Our aim will be to develop a brief set of questions, characteristics, proxy measures, or indices that could be readily and efficiently determined at intake/enrollment in a program and which would assist in identifying a chronically homeless client. Do you have any suggestions of possible questions/characteristics that should be included in this index?
8. Because this program may be one of several serving the participants, either concurrently or sequentially, do you think the performance measures used or proposed can isolate the effects of this program? Do you have any suggestions for dealing with this?

Discussion Guide for Initial Visits to
DHHS Homeless-Serving Programs:
Runaway and Homeless Youth Program

Interviewee:
Date of Interview:

Background on the Program and Services (Basic Center Program) 

1. Please provide a brief history of the programs origins and evolution.
2. What is the most recent annual budget for the program? What are the budget requests for the next several out years? Is all the money allocated by formula? If other distribution means are used, please describe.
3. Please describe the general structure of the program:
   1. What is the role of the federal government?
   2. What is the role for states?
   3. Who are the grantees? What is the distribution among states, cities and counties, CBOs, and other organizations?
   4. How common is it for a grantee to provide all or most of the programs services as opposed to using subcontractors/sub-grantees? What is the nature and distribution of the subcontractors/sub-grantees?
4. Please provide a copy of the GPRA plan and outcome measures for this program and address the following questions:
   1. Are all the GPRA measures of equal importance to the agency? If not, please rank them.
   2. Are there any other performance measures beyond the GPRA measures that you currently use? If so, what are they, how are they tracked, and how do they rank compared to the GPRA measures?
5. Are there any national, state, or local evaluations of the program? If so, would you characterize these evaluations as process studies, outcome studies, or net impact evaluations? How do you use the evaluations in measuring performance of your grantees? How do the evaluations tie in to your GPRA work? Can you provide us with several examples of evaluations, including some that are good and some that are poor in quality?
6. How do the states distribute their funds? Do they sometimes run programs themselves? Do they sponsor competitions and/or do they distribute the funds to county and local government?
7. How competitive is the program? What proportion of grant applications are funded? What differentiates those funded from those not funded?
8. Once recruited, how do participants flow through the program (i.e., from recruitment, through intake and assessment, and into program services)?
9. Please briefly describe the main types of services and activities participants receive (i.e., health care services; mental health services; substance abuse services; employment and training services; help resolving housing problems; other support services)  [note: indicate whether all participants receive a specific service or only some participants receive the service]?
10. Do local programs typically partner with other agencies to provide services to participants (i.e., refer participants to other agencies for services)? If so, who are the other organizations and what services do they provide? How does the partnering work? Is there a subcontract, formal referral, informal referral, or some other approach?
11. At what point in the process is an individual considered a program participant? Are program participants formally terminated from the program, and if so, when? How long do participants typically stay enrolled in the program (i.e., range/average length of involvement)? Is there a problem with attrition (before participants complete program services)?
12. What do you believe to be the major impacts/effects of the program on participants? (Indicate the outcomes and the levels typically achieved)

Performance Measurement and Information Flow:

1. Please provide us with copies of all data collection requirements and forms for grantees. If you have reports from previous years, please provide us with copies. What data do you require beyond the GPRA outcomes, and why do you require it?
2. Explain all the uses you make of data reported by grantees, including rewards, sanctions, use in future funding, completing GPRA reports, etc.
3. Do you tie the evaluation data in any way to the GPRA data in assessing the program?
4. Has the federal office implemented a standardized automated data system across states and program grantees to collect performance data? [If yes, obtain documentation on the data system, such as a copy of participant forms.] Alternatively, have grantees developed their own automated data systems? If yes, please provide a brief description of the various types of systems in use (e.g., is there great diversity in the types of systems in use)?
5. How satisfied are program administrators with the existing performance monitoring and/or data system? Are there ways in which the performance monitoring or data systems might be improved? Is there any planned or ongoing effort to change either the types of performance information collected or the way in which these data are collected?
6. What types of information would your agency ideally want to be able to measure regarding program performance (e.g., process and outcome measures)? To what extent would it be possible for states/grantees to report on these measures?
7. As part of this study, we are exploring possible methodologies for identifying chronically homeless individuals. Our aim will be to develop a brief set of questions, characteristics, proxy measures, or indices that could be readily and efficiently determined at intake/enrollment in a program and which would assist in identifying a chronically homeless client. Do you have any suggestions of possible questions/characteristics that should be included in this index?
8. Because this program may be one of several serving the participants, either concurrently or sequentially, do you think the performance measures used or proposed can isolate the effects of this program? Do you have any suggestions for dealing with this?

Discussion Guide for Telephone Interviews with Administrators of HADS

Questions/Topics to Discuss During Telephone Interviews with HADS Registry Systems

 

1. What is name of the system?
2. What year was the system developed?
3. Why was the system developed?
4. Who (what agency) maintains the system)?
5. Are there other partnering agencies (and if so, what agencies)?
6. Who is the data maintained on (e.g., what group of homeless individuals)?
7. How many individuals have been entered into the system since its inception?
8. What data items do you collect on each individual (e.g., demographics, services received, outcomes)?
   1. Could you send a list of data elements?
   2. Could you provide a copy of manual forms used to collected data?
9. At what point(s) are data collected from participants? (e.g., at intake/assessment, at regular intervals during participants involvement, at case closure, after case closure)?
10. Who (what agencies or programs) collect the data?
11. Who (what agencies or programs) enters data into the registry system?
12. Into what software is the data entered (e.g., proprietary software, off-the-shelf software)?
13. Is the system accessible via the Internet (if so, to who is it accessible and at what web address)?
14. What is done with the data once it is entered into the system?
    1. Is it used for reporting purposes?
    2. Could you provide a copy of sample reports generated by the system?
15. What have been the overall benefits of the system?
16. What does it cost to operate the system? Where does funding come from to cover these costs?
17. Did you run into any particular challenges/problems in developing or implementing the system?
18. Would it be okay if we were to visit to learn more about the system and its uses? If so, when would it be convenient for us to visit?
19. Could you please send any additional background documentation on the system to us?