Toward Understanding Homelessness: The 2007 National Symposium on Homelessness Research. Changing Homeless and Mainstream Service Systems: Essential Approaches to Ending Homelessness. Evaluation Processes

09/01/2007

Communities have established a range of practices to measure progress and to influence further change, many of which rely on the infrastructure components described above. The primary methods include program-level evaluation, performance-monitoring and funding tied to performance, and benchmarking system progress. Many communities and providers note that having a process in place to measure their actions and results holds them more accountable, and therefore makes them work harder to be productive so they will be able to demonstrate results.

Program-Level Evaluation and Performance Monitoring and Accountability

Client records tracked in a longitudinal database, such as an HMIS, can be used systematically to understand program performance. The results can identify effective program practices or low-performing programs that need technical assistance to improve their performance. Of course, a communitys ability to use HMIS information in this way is only possible if the database contains fields for the relevant outcomes and if providers are diligent about collecting and entering the relevant information.

The results of performance monitoring can be used to direct clients to the programs that appear to be most successful for people with similar characteristics and issues. They can also be used to direct funding to successful programs and divert limited community resources from less successful efforts. Communities across the country are putting these and other strategies in place in as part of their efforts to reduce or end homelessness and to understand the processes of system change that can help them reach that ultimate goal. The earlier discussion of Philadelphias approach provides one example, in addition to those below.

Columbus has implemented advanced processes for analyzing program effectiveness with HMIS data and uses the results to influence program funding. To support its process, the Community Shelter Board (CSB) has developed data quality assurance standards for all funded agencies to ensure that the community has reliable, complete data on which to base decisions. CSB has also developed comprehensive program standards and performance expectations for each program type, and incorporates these expectations into each agencys contract. Performance measures pertain to the number of clients served, average length of stay, housing and income outcomes at exit, return to homelessness, client movement toward agreed-upon goals, direct client assistance utilization, occupancy rates, housing stability and retention, and efficient use of funding resources (average per client costs). CSB clearly communicated to agencies that these outcomes were a priority in the homeless system. It identified measures to support performance-based funding and put them in place throughout the system of homeless assistance services. Over the years, tracking performance has helped to fuel program-level change to support these goals.

The Michigan Measurement Project was established in 2006 to develop a sophisticated program outcomes measurement system to track intermediate and long-term client outcomes by program type. Once implemented through the statewide HMIS, aggregate performance data will be viewable through the HMIS at the program, agency, CoC, and statewide levels. Programs can compare their own results to those of other like programs throughout the state to assess their own effectiveness and to identify strategies for improvement. CoCs and state agencies can use the outcome measures to inform community planning and resource allocation processes. Because it is a state-level activity, the Michigan Measurement Project is more focused on identifying promising program practices and benchmarking program progress than local evaluation processes might be. The latter might focus more on performance monitoring and performance-based funding.

The model of measuring system change at the program level is also being carried to the federal level (Khadduri, 2005). HUD is presently reconfiguring its Annual Progress Report to improve the quality of information it receives from continuum-of-care grantees on the activities and outcomes of HUD-funded programs in relation to its national goal and other related objectives. To reflect the diverse goals of homeless assistance programs, particularly in light of local system change efforts, HUD is contemplating establishing a range of performance measures from which funded programs could select when they apply for HUD funding. HUD would then use the selected measures to monitor each grantees accomplishments. HUD can integrate the program-level results with two other primary sources to measure national progress toward the goal of ending chronic homelessness. First, HUD is analyzing HMIS data from a sample of communities across the country to produce Annual Homeless Assessment Reports on the extent and nature of homelessness and use of homeless services. Second, HUD asks CoCs to report numbers of sheltered and unsheltered persons in their annual CoC applications. Together, these sources help HUD understand better how homelessness is changing over time and suggest how programs funded through CoC Care grants are contributing to these changes.

System-Level Evaluation

Communities are also instituting processes to track their own effectiveness in achieving system change for the ultimate purpose of ending homelessness. Establishing an evaluation framework forces a community to set deliberate change goals and to identify strategies to accomplish them, from which it can easily document progress related to its multi-year action plan. As noted earlier, evaluating the process may actually promote progress itself. The change goals and strategies become the outcomes and indicators of the effects of changing systems that are tracked over time. A comprehensive evaluation framework is likely to include process outcomes (what indicators will help the community know if it has completed the strategies that it anticipates are needed to effect system change), program outcomes that can be used to guide a program-level evaluation and performance-based funding process (what client outcomes are needed at the program level in order to achieve the system outcomes), and system outcomes (the domains that the community hopes will be impacted as a result of the efforts). If all of these components are in place, a community will be able to observe whether its intended system changes have occurred and if they are making an impact. If the expected effects are not observed, the community will also have data to help indicate whether it failed to meet its outcomes because plans were inadequately carried out, or whether the strategies the community thought would help were insufficient. The information can be used to improve their process over time, and the community can share effective strategies and/or pitfalls with others trying to accomplish similar things.

The Chicago continuum of care developed a series of system measures to assess progress in ending homelessness, including indicators of the number of people (overall and chronically homeless) who present for homeless assistance each year, the number of days it takes to help someone presenting with a housing crisis return to permanent housing placement, rates of permanent housing retention, and rates of recidivism. Program outcomes have been defined for each program type to help set common expectations for what is expected from each part of the system and to guide resource allocation. The CoC also developed process and efficiency measures, such as annual projections of units and services slots by year (some program types will increase, some will decrease over time), the vacancy rates of residential programs, user satisfaction rates, and indicators related to increasing resources for appropriate interventions. Chicago has been publishing semi-annual State of the Plan reports (available at www.cityofchicgao.org) that share accomplishments and annual outputs related to unit conversion and development targets as well as performance on the system measures.

Various formal methods exist to evaluate changes in system connectedness and integration, including changes in how people and agencies relate to each other, changes in how clients are referred among agencies, and changes in how funding does or does not flow among agencies. In the homeless arena, some of these methods were used in evaluating the Program on Chronic Mental Illness that began in the mid-1980s with support from the Robert Wood Johnson Foundation and HUD, and also the ACCESS demonstrations of the early 1990s (Morrissey et al., 1994, 2002; Rosenheck et al., 1998, 2001, 2002, 2003b). Modifications of the same methods are now being used to assess changes in system integration associated with the Collaborative Initiative on Chronic Homelessness (Greenberg & Rosenheck, 2006). These methods are available for use by communities seeking formal quantitative documentation of system change.

A community or researcher could also apply one or both frameworks presented in the beginning of this paper to measure process aspects of system change: the five indicators of change from Laying a New Foundation, and the five stages of integration. Laying a New Foundation indicators of change in money, ideas, habits, power, and skills/technology can be measured as process indicators. For instance, a community that sets a goal to shift system resources from shelters to permanent housing could track the percentage of resources going to each component of its homeless assistance system over time. Alternatively, a community could track the number of units/service slots in each program area over time. Changes in skills and habits could be measured by tracking first whether staff have acquired the skills and tools they need to conduct business differently, and second whether the business practices have actually changed. For instance, are people being placed within 14 days or are staff continuing to work with clients using the old patterns of doing business? These measures need to be constructed locally depending on the ways in which the local systems need to change; however, the framework can provide useful categories to classify and set expectations for change.

The stages of integration and changes in money, ideas, habits, skills, and power are or should be interactive. That is, as a community progresses toward more collaboration and coordinated community response, one expects these changes in process to produce changes in money, power, ideas, etc. But it also works the other way, as changes in money, power, and so on can push systems to change more and cause more stakeholders to join the system change bandwagon. In addition to the usual approach involving qualitative methods to assess changes in stages of services and systems integration, the formal methods referenced above can be used to document changes in the flow of clients, ideas, and money. Some changes, such as changes in money or new housing units, are relatively easy to document (e.g., Burt & Anderson 2005, 2006).

Communities could commit themselves to assessing the extent of change in the level of integration across all the agencies in a whole system as well as between any two agencies or among any three or more agencies within a particular agency or system. Also relevant are the isolated or integrated activities of various divisions within large umbrella public agencies. For example, many Departments of Human Services include divisions responsible for income maintenance (TANF, food stamps), child care, child welfare, and sometimes mental health and homeless assistance. It is all too common that these divisions do not work together at all, despite all existing under a common roof.

One can use the framework to prescribe, as well as describe, changes in various systems that are expected to promote the goal of ending homelessness. A community can assess its initial or starting point of integration and set explicit goals as to where, along the continuum of integration stages, the systems should be in one year, two years, five years, and so on. In doing so, it is essential to realize that it would be very unusual for all the agencies and organizations in a community to be at the same place at the same time in this five-level framework. Rather, what usually happens is that some parts of a potential system begin moving toward increased integration early on and, if their progress is viewed as useful, begin to bring other elements on board and to expand the scope of activities. It is therefore most fruitful to use this framework to characterize movement and change rather than a steady state or a comprehensive overview. A community does not need all potentially relevant agencies on board at the start, but as ending homelessness will not happen without the substantial commitment of public resources, at least some major public agencies need to be committed at the outset for a community to be able to speak realistically about embarking on a campaign to end homelessness.

An illustrative example for a community deciding to work toward ending family homelessness is shown in Exhibit 3. The matrix in Exhibit 3 should really be three-dimensional, as the issue with integration is how one agency or system works with one or more other agencies or systems. This multi-agency nature of integration is reflected in the last column of Exhibit 3, where the coordinated community response cell is shown as including participation of six different systems.

Exhibit 3
Sample Framework for Assessing and Advancing Stages of Integration Across Community Partners to
Implement a Plan to End Family Homelessness
Community Partners Isolation Communication Coordination Collaboration Coordinated Community Response
Family Homeless Service Providers   Today most talk with each other, refer to each other 6 months beginnings of agreed-upon specialization,  partnering for specialized services Year 1 Collaborative project among several family homeless providers Year 5 Centralized intake and triage for housing placement; scaleable case management supported by full range of integrated service partners and pool of vouchers and flexible resources to meet family housing and service needs
Child Welfare Today Year 1 communications established with family homeless providers, agree to work on reducing distrust, antagonistic relationships Year 2 develop system to identify common families and triage methods, identify child welfare resources to assist homeless families without threat of removal Year 3 Service Integration Pilot , family reunification project includes homeless providers
Family Court Today Year 1 part of communications that include child welfare and homeless providers Year 2 work on court standards and expectations for working with homeless families
Workforce Development Agencies Today Year 1 Homeless Planning Partner Year 2 One-Stop Career Center staff co-located at family homeless programs  
Public Housing Authority   Today Year 1 Streamlined application for PHA resources Year 2 Dedicated vouchers for homeless families, support systems in PHA buildings to prevent homelessness, management of locally-supported short- and medium-term vouchers
TANF/food stamps/Medicaid Agency Today Year 1 Benefits Eligibility Analysis Year 2 Automated Benefits  Screening by Homeless Providers Year 5 Automated Benefits Enrollment by Homeless Providers Long-term System of Care Pooled Resources Approach
Mental Health and Substance Abuse Agencies Today Year 1 convince to come to table, recognize shared clients Year 2 negotiate access to specialized resources, develop partnering techniques Year 3 develop dedicated funding streams  
State Affordable Housing/Subsidies Funders (Housing Finance/RedevelopmentAgencies, Legislature) Today Year 1 start to make contacts Year 2 work on possible programs, legislation; convince with performance results Year 3 beginning of new resource availability Long-term new dedicated housing affordability resources to reduce need

Similar assessments of the current stage of integration can occur at subsequent time intervals, and the community can use the results to help assess whether the past actions were effective in meeting change goals. This type of evaluation does not replace the need to assess whether an increasingly integrated system is reducing homelessness or positively affecting the problem that the community is targeting. Both the formative (process) and summative (end result) measures are required for communities to understand which strategies are most effective in meeting identified needs.

Some communities may also find it useful to think of how they might modify the stages of change framework used in the substance abuse treatment field for use in documenting community change. The stages of change framework classifies a persons willingness and progress in addressing a substance addiction. Such an approach recognizes the importance of assessing where an individual is relative to his/her understanding of a need to change and his/her commitment to the change process and identifies a predictable sequence of stages through which an individual passes during the process: precontemplation, contemplation, determination, action, maintenance, and relapse prevention (Prochaska et al, 1994). Programs that use a stages of change approach assess and respect where an individual is in that process and encourage and motivate movement. The strategies and actions that a treatment professional might use to engage an individual in treatment or the change process will vary depending on a determination of the persons current stage of acknowledgement and engagement. The stages in the addictions scheme are used over time to benchmark progress and regression. If one is willing to think of a whole community in terms of its acknowledgement of the need to change and its willingness to take steps to change, this model might be applicable. It would certainly be applicable to measuring changes in individual homeless people as they move toward leaving homelessness.

View full report

Preview
Download

"report.pdf" (pdf, 455.28Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®