Core Performance Indicators for Homeless-Serving Programs Administrated by DHHS

Potential Core Performance Measures for Homeless-Specific Service Programs

[ Main Page of Report | Contents of Report ]

Contents

  1. Considerations and Constraints on Developing a Common Set of Performance Measures
  2. Suggested Core Performance Measures
  3. Conclusions

This chapter identifies a potential core set of performance measures that could be common across homeless-serving programs of DHHS. The measures — including both process and outcome measures — suggested in this chapter are intended to enhance DHHS tracking of services and outcomes for homeless individuals served in DHHS homeless-serving and non-homeless-serving programs. This chapter includes the following sections: (a) discussion of several of the constraints in creating core performance measures; and (b) identification of a potential core set of homeless measures and discussion of technical implications for incorporating such measures into the current performance reporting approaches utilized by DHHS.

A. Considerations and Constraints on Developing a Common Set of Performance Measures

Using the material and analyses conducted under earlier study tasks, the focus of this chapter is on offering a set of suggested performance measures that could be common and useful across homeless-serving programs of DHHS. In our presentation and analysis of these measures, we have attempted to differentiate between performance measures that are possible from current reporting approaches and those derived from HADS operations, operations needed to collect and aggregate the data, and quality and uses of the data. In developing these measures, we took into consideration the following important factors:

In proposing a set of core performance measures for the four homeless-serving programs that are the focus of this study, the findings from our earlier review of each program and its current performance measurement system catalogue constraints for development of a common set of performance measures that cut across the programs. Perhaps most important, our earlier analysis of the four homeless-serving programs indicated that there are substantial cross-program differences that complicate efforts to develop similar performance measures and systems for collecting data (see Chapter 2 for more detailed discussion of cross-program differences and for a chart comparing the four homeless-serving programs). For example:

Given the variation in the structure of these four programs, it is not surprising that the four homeless-serving programs have adopted quite different approaches to information collection, performance measurement, and evaluation (see Exhibit 2-2 earlier for specific measures used by each program). With respect to GPRA measures, three of the four programs have explicit measures; there are no GPRA measures specific to HCH. GPRA measures apply to the BPHC’s Health Centers Cluster of programs as a whole, of which HCH program is part.(15) The measures used for the three other programs include both process and outcome measures. The Treatment for Homeless Persons Program has outcome-oriented GPRA measures, as well as a data collection methodology (featuring intake and follow-up client surveys) designed to provide participant-level data necessary to produce the outcome data needed to meet reporting requirements. RHY and PATH employ mostly process-oriented GRPA measures.

Reliability and quality of data collected and submitted to federal offices varies by program. In addition, intensity and duration of participant involvement in the four homeless-serving programs ranges considerably across and within programs, with implications for performance measurement: short or episodic involvement (such as the involvement in some participants in RHY and HCH programs) limit opportunities for collection of data from participants.

In Chapter 2, we concluded that it would be both a difficult and delicate task to develop a common set of performance measures across the four homeless-serving programs. We noted that the willingness and ability of programs to undertake change (e.g., incorporate new outcome-oriented GPRA measures) is uncertain and the changes in how programs collect data and report on performance would require substantial efforts on the part of agency officials and programs. Hence, in specifying performance measures, it is important to be sensitive to the substantial cross program differences and the constraints that program administrators (at the federal, state, and local levels) face in making changes to how they collect and report on program performance.

B. Suggested Core Performance Measures

Despite the difficulties and constraints in developing a core set of performance measures, our review of the performance measurement systems in existence across the four programs also indicates potential for both enhancement and movement toward more outcome-oriented measures. For example, the general approach to performance measurement used within the Treatment for Homeless Persons Program — which features pre/post collection of participant-level data and outcome-oriented measures — provides a potential approach that could be applicable to the other three programs (as well as other non-homeless-serving programs operated by DHHS). In suggesting a potential set of core performance measures cutting across these four homeless-serving programs, it is important to consider where the four programs intersect with respect to program goals/objectives for the homeless individuals being served. From this commonality of goals arises the potential for a core set of measures (with the recognition, however, that each program will also likely require additional measures specific to differing objectives and service offerings). Of critical important to our efforts to suggest core measures, all four of the programs are aimed at (1) improving prospects for long-term self-sufficiency, (2) promoting housing stability, and (3) reducing the chances that individuals will become chronically homeless.(16) In addition, the four programs (some more than others) also stress addressing mental and physical health concerns, as well as potential substance abuse issues.

Based on the common objectives of these four programs, we suggest a core set of process and outcome measure that could potentially be adapted for use by the four homeless-service programs (see Exhibit 4-1). We suggest selection of the four process measures, which track numbers of homeless individuals (1) contacted/outreached, (2) enrolled, (3) comprehensively assessed, and (4) receiving one or more core services. We then suggest selection of several outcome measures from among those grouped into the following areas: (1) housing status, (2) employment and earnings status, and (3) health status. In addition, we have suggested a several additional outcome measures that could be applied to homeless youth.

Exhibit 4-1:
Potential Core Performance Measures for DHHS Homeless-Serving Programs
Type of Measure Core Performance Measure When Data Item Could Be Collected Comment
**PROCESS MEASURES**
Process # of Homeless Individuals Contacted/Outreached At first contact with target population  
Process # of Homeless Individuals Enrolled At time of intake/ enrollment or first receipt of program service  
Process Number/Percent of Homeless Individuals Enrolled That Receive Comprehensive Assessment At time of initial assessment May include assessments of life skills, self-sufficiency, education/training needs, substance abuse problems, mental health status, housing needs, and physical health
Process Number/Percent of Homeless Individuals Enrolled That Receive One or More Core Services At time of development of treatment plan, first receipt of program service(s), or referral to another service provider Core services include:
  • Housing Assistance
  • Behavioral Health Assistance (Substance Abuse/Mental Health Treatment)
  • Primary Health Assistance/Medical Treatment
**OUTCOME MEASURES — HOUSING STATUS**
Outcome — Housing Number/Percent of Homeless Individuals Enrolled Whose Housing Condition is Upgraded During the Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
Possible upgrade categories:
  • Street
  • Emergency Shelter
  • Transitional Housing
  • Permanent Housing
Outcome — Housing

Number/Percent of Homeless Individuals Enrolled Who Are Permanently Housed During the Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
 
Outcome — Housing Number/Percent of Homeless Individuals Enrolled Whose Days of Homelessness (on Street or in Emergency Shelter) During the Past Month [or Quarter] Are Reduced
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • HADS systems may provide useful data on shelter use (but not street homelessness)
**OUTCOME-MEASURES — EARNING/EMPLOYMENT STATUS**
Outcome — Earnings Number/Percent of Homeless Individuals Enrolled with Earnings During the Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • UI quarterly earnings data (matched using SSN) could be useful — though data lags, potential costs, and confidentiality issues
Outcome — Earnings Number/Percent of Homeless Individuals Enrolled with Improved Earnings During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • UI quarterly earnings data (matched using SSN) could be useful — though data lags, potential costs, and confidentiality issues
Outcome — Employment Number/Percent of Homeless Individuals Enrolled Employed 30 or More Hours per Week
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • Hours threshold could be changed (20+ hours; 35+ hours); hours worked could be for week prior to survey or avg. for prior month or quarter
  • UI quarterly wage data not helpful (hours data not available); so follow-up survey probably needed
Outcome — Employment Number/Percent of Homeless Individuals Enrolled with Increased Hours Worked During the Past Month [Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • UI quarterly wage data not helpful (hours data not available); so follow-up survey probably needed
**OUTCOME MEASURES — HEALTH STATUS**
Outcome — Substance Abuse Number/Percent of Homeless Individuals Enrolled and Assessed with Substance Abuse Problem That Have No Drug Use the Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • Drug screening could be used
Outcome — Physical Health Status Number/Percent of Homeless Individuals Enrolled Assessed with Physical Health Problem That Have Good or Improved Physical Health Status During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • May be difficult to objectively measure “good or improved”
Outcome — Mental Health Status Number/Percent of Homeless Individuals Enrolled Assessed with Mental Health Problem That Have Good or Improved Mental Health Status During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • May be difficult to objectively measure “good or improved”
**OUTCOME MEASURE — YOUTH-ONLY**
Outcome — Family Reunification Number/Percent of Homeless & Runaway Youth Enrolled That Are Reunited with Family During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
  • Reunification may not always be an appropriate outcome — and it is often hard to know when it is
Outcome —   Attending School Number/Percent of Homeless Youth Enrolled That Attended School During Past Month [or Quarter]
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
 
Outcome — Completing High School/GED Number/Percent of Homeless Youth Enrolled That Complete High School/GED During Past Quarter
  • At intake/enrollment
  • 3, 6, and/or 12 months after point of enrollment
  • At termination/exit
 

With regard to housing outcomes, we have identified three potential outcome measures intended to track (1) changes in an individual’s housing situation along a continuum (from living on the street and in emergency shelters to securing in permanent housing), (2) whether the homeless individual secures permanent housing, and (3) days of homelessness during the preceding quarter (or month). It should be noted with regard to housing outcomes, that although the four homeless-serving programs focus primarily on other non-housing related goals and services (e.g., improving mental or physical health status, reducing/eliminating substance abuse, reuniting runaway youth with their families), that housing outcomes for homeless individuals are of paramount importance. Housing outcomes are appropriate to consider for programs focused on homelessness even when their primary goals may be focused on improving mental health status or physical health status.

Two earnings measures are identified — one that captures actual dollar amount of earnings during the past quarter (or month) and a second measure that captures whether an individual’s earnings have improved. Two employment measures are also identified — one relating to whether the individual is engaged in work 30 or more hours per week and another that measures whether hours of work have increased. Three health-related measures are offered, focusing on use of drugs, improvement in physical health status, and improvement in mental health status. Finally, three measures are offered that are targeted exclusively on youth (though the other outcome measures would for the most part also be applicable to youth): (1) whether the youth is reunited with his/her family, (2) whether the homeless youth is attending school, and (3) whether the homeless youth graduates from high school or completes a GED.

A pre/post data collection approach is suggested with respect to obtaining needed performance data — for example, collecting data on housing, health, and substance abuse status of program participants at the time of intake/enrollment into a program and then periodically tracking status at different points during and after program services are provided (i.e., at termination/exit from the program and/or at 3, 6, or 12 months after enrollment). Collection of data on homeless individuals at the point of termination can be problematic because homeless individuals may abruptly stop coming for services. The transient nature of the homeless population can also present significant challenges to collecting data through follow-up surveys/interviews after homeless individuals have stopped participating in program services (e.g., at 12 months after enrollment).

Given difficulties of tracking homeless individuals over extended periods (and particularly after individuals’ termination from programs), the extent to which existing administrative data can be utilized could increase the proportion of individuals for which it is possible to gather outcome data (at a relatively low cost). Probably the most useful source in this regard is quarterly unemployment insurance (UI) wage record data, which can be matched by Social Security number (though releases are required and it may also be necessary to pay for the data). UI wage withholding data provides the opportunity to track earnings on a quarterly basis (from covered employers) and, for example, examine how earnings may change from quarter to quarter and potential effects of program involvement on workforce participation and economic self-sufficiency.

A second potential source of administrative data that may have some potential utility for tracking housing outcomes are HADS system maintained by many states and/or localities. As noted in Chapter 3, HADS systems are not used principally for measuring program performance or outcomes — though have the capability to provide analyses of length of stay. The HADS principally serve as registry systems that facilitate tracking of program participant characteristics, services received, length of stay, and movement within emergency and transitional housing facilities. Such systems may provide useful data for tracking use of emergency and transitional housing, as well as chronic homelessness — though are limited for purpose of determining housing status once an individual leaves emergency or transitional housing (i.e., on the street or in permanent housing).

Finally, in terms of tracking self-sufficiency outcomes, data sharing agreements with state and local welfare agencies may provide possibilities for tracking dependence on TANF, food stamps, general assistance, emergency assistance, and other human services programs.

C. Conclusions

The process and performance measures outline in this final report are suggestive of potential measures that could cut across the four homeless-serving programs. It is recommended that careful thought be given to the development and implementation of such measures so that programs are not burdened by large numbers of overly complicated performance measures. Each measure added will likely require program staff to make changes in data collection forms, procedures, and automated data systems, as well as likely impose added burden and costs on program staff and participants. However, given the increasing emphasis on measurement of program performance in recent years by Congress and the potential for performance data to provide valuable feedback for enhancing service delivery, it is critical to identify potential ways in which programs can better track participant outcomes — particularly, changes in status (e.g., housing situation or earnings) from the time of entry into homeless-serving programs through termination and beyond.

Building on outcome measures suggested in this report and moving beyond the specific programmatic outcomes for participants in the four DHHS homeless-serving programs that are the focus of this study, it may be possible down the road to introduce (1) experimental designs for measuring “net impacts” of program services and (2) “system-wide” measures that communities may be able to use to gauge the overall success of their efforts to counter problems associated with homelessness. Such experimental designs could employ some of these same outcome measures, but compare outcomes (e.g., whether days of homelessness are reduced or whether labor force attachment and earnings increase) for individuals receiving program services versus similar outcomes for a randomly assigned control group of homeless individuals (not receiving services). Introduction of “system-wide” measures could provide the opportunity for exploring the wider potential effects of a group of or all homeless services within a particular locality (as well as other contextual factors, such as local economic conditions and loss of affordable housing). Such system-wide measures would not be used to hold individual programs accountable for achievement of specified outcomes, but rather enable state and local decision-makers (e.g., a mayor of a large metropolitan area) to address more expansively questions about the local homeless situation, such as “is the problem of chronic homelessness intensifying in the community” or “is the community making a dent in the number of homeless individuals on the streets and living in emergency shelters each night” or “to what extent is the community addressing its general homeless problem.”

Endnotes

(15) HCH is clustered with several other programs, including Community Health Centers [CHCs], Migrant Health Centers, Health Services for Residents of Public Housing, and other community-based health programs.

(16) At the same time, each program has more specific goals which relate to the populations served and related to its original program intent — for example, RHY’s BCP component has as one of its goals family reunification (when appropriate); HCH aims to improve health care status of homeless individuals; PATH aims to engage participants in mental health care services and improve mental health status; and the Treatment for Homeless Persons program aims at engaging participants in substance abuse treatment and reducing/eliminating substance abuse dependency.


Where to?

Top of Page | Contents
Main Page of Report | Contents of Report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)