Core Performance Indicators for Homeless-Serving Programs Administered by the U.S. Department of Health and Human Services. Discussion Guide for Interviews with Administrators and Staff of Four Homeless-Serving Programs

09/01/2003

Discussion Guide for Initial Visits to
DHHS Homeless-Serving Programs:
Addictions Treatment for Homeless Persons Program

Interviewee:
Date of Interview:

Background on the Program and Services 

  1. .Please provide a brief history of the programs origins and evolution.
  2. What is the most recent annual budget for the program? What are the budget requests for the next several out years? Is all the money allocated via SGAs, such as the one on the web, or is some of the money also distributed by formula? If other distribution means are used, please describe.
  3. Please describe the general structure of the program:
    1. What is the role of the federal government?
    2. Is there a specific role for states, or is their role limited to their role as grantees?
    3. Who are the grantees? What is the distribution among states, cities and counties, CBOs, and other organizations?
    4. How common is it for a grantee to provide all or most of the programs services as opposed to using subcontractors/sub-grantees? What is the nature and distribution of the subcontractors/sub-grantees?
  4. The agencys GPRA plan and outcome measures are attached to the SGA. Please answer the following questions:
    1. Are all the GPRA measures of equal importance to the agency? If not, please rank them.
    2. Are there any other performance measures beyond the GPRA measures that you currently use? If so, what are they, how are they tracked, and how do they rank compared to the GPRA measures?
  5. The SGA indicates that grantees are required to conduct local evaluations. Would you characterize these evaluations as process studies, outcome studies, or net impact evaluations? How do you use the local evaluations in measuring performance of your grantees? How do the local evaluations tie in to your GPRA work? Can you provide us with several examples of local evaluations, including some that are good and some that are poor in quality?
  6. Please provide annual enrollment and budget data for the past three years for the program and current year budget and target for enrollment.
  7. On page 6 of the SGA, the target population for the SGA is described. Does this apply to all uses of the programs funds? Are some subgroups considered of higher priority than others?
  8. How competitive is the program? What proportion of grant applications are funded? What differentiates those funded from those not funded?
  9. Once recruited, how do participants flow through the program (i.e., from recruitment, through intake and assessment, and into program services)?
  10. Please briefly describe the main types of services and activities participants receive (i.e., health care services; mental health services; substance abuse services; employment and training services; help resolving housing problems; other support services)  [note: indicate whether all participants receive a specific service or only some participants receive the service]?
  11. Do local programs typically partner with other agencies to provide services to participants (i.e., refer participants to other agencies for services)? If so, who are the other organizations and what services do they provide? How does the partnering work? Is there a subcontract, formal referral, informal referral, or some other approach?
  12. At what point in the process is an individual considered a program participant? Are program participants formally terminated from the program, and if so, when? How long do participants typically stay enrolled in the program (i.e., range/average length of involvement)? Is there a problem with attrition (before participants complete program services)?
  13. What do you believe to be the major impacts/effects of the program on participants? (Indicate the outcomes and the levels typically achieved)

Performance Measurement and Information Flow:

  1. Please provide us with copies of all data collection requirements and forms for grantees. If you have reports from previous years, please provide us with copies. What data do you require beyond the GPRA outcomes, and why do you require it?
  2. Explain all the uses you make of data reported by grantees, including rewards, sanctions, use in future funding, completing GPRA reports, etc.
  3. Do you tie the local evaluation data in any way to the GPRA data in assessing the program?
  4. Has the federal office implemented a standardized automated data system across program grantees to collect performance data? [If yes, obtain documentation on the data system, such as a copy of participant forms.] Alternatively, have grantees developed their own automated data systems? If yes, please provide a brief description of the various types of systems in use (e.g., is there great diversity in the types of systems in use)?
  5. How satisfied are program administrators with the existing performance monitoring and/or data system? Are there ways in which the performance monitoring or data systems might be improved? Is there any planned or ongoing effort to change either the types of performance information collected or the way in which these data are collected?
  6. What types of information would your agency ideally want to be able to measure regarding program performance (e.g., process and outcome measures)? To what extent would it be possible for states/localities to report on these measures?
  7. As part of this study, we are exploring possible methodologies for identifying chronically homeless individuals. Our aim will be to develop a brief set of questions, characteristics, proxy measures, or indices that could be readily and efficiently determined at intake/enrollment in a program and which would assist in identifying a chronically homeless client. Do you have any suggestions of possible questions/characteristics that should be included in this index?
  8. We were a bit confused by Appendix B on the agencys GPRA plan, Could you please walk us through it and explain how you are implementing it for your program?
  9. Appendix C provides an OMB approved client survey instrument. Could you provide us a description of how the data are collected and used?
  10. Because this program will usually be one of several serving the participants, either concurrently or sequentially, do you think the performance measures used or proposed can isolate the effects of this program? Do you have any suggestions for dealing with this?

Discussion Guide for Initial Visits to
DHHS Homeless-Serving Programs:
Health Care for the Homeless

Interviewee:
Date of Interview:

Background on the Program and Services 

  1. Please provide a brief history of the programs origins and evolution.
  2. What is the most recent annual budget for the program? What are the budget requests for the next several out years? How are funds allocated/distributed to states/local grantees (e.g., formula, competitive process)? If other distribution means are used, please describe.
  3. Please describe the general structure of the program:
    1. What is the role of the federal government?
    2. What is the role for states?
    3. Who are the grantees? What is the distribution among states, cities and counties, CBOs, and other organizations?
    4. How common is it for a grantee to provide all or most of the programs services as opposed to using subcontractors/sub-grantees? What is the nature and distribution of the subcontractors/sub-grantees?
  4. Please provide a copy of the GPRA plan and outcome measures for this program and address the following questions:
    1. Are all the GPRA measures of equal importance to the agency? If not, please rank them.
    2. Are there any other performance measures beyond the GPRA measures that you currently use? If so, what are they, how are they tracked, and how do they rank compared to the GPRA measures?
  5. Are there any national, state, or local evaluations of the program? If so, would you characterize these evaluations as process studies, outcome studies, or net impact evaluations? How do you use the evaluations in measuring performance of your grantees? How do the evaluations tie in to your GPRA work? Can you provide us with several examples of evaluations, including some that are good and some that are poor in quality?
  6. If any HCH funding is distributed to states, how do the states distribute their funds? Do they sometimes run programs themselves?
  7. How competitive is the program? What proportion of grant applications are funded? What differentiates those funded from those not funded?
  8. Once recruited, how do participants flow through the program (i.e., from recruitment, through intake and assessment, and into program services)?
  9. Please briefly describe the main types of services and activities participants receive (i.e., health care services; mental health services; substance abuse services; help resolving housing problems; other support services)  [note: indicate whether all participants receive a specific service or only some participants receive the service]?
  10. Do local programs typically partner with other agencies to provide services to participants (i.e., refer participants to other agencies for services)? If so, who are the other organizations and what services do they provide? How does the partnering work? Is there a subcontract, formal referral, informal referral, or some other approach?
  11. At what point in the process is an individual considered a program participant? Are program participants formally terminated from the program, and if so, when? How long do participants typically stay enrolled in the program (i.e., range/average length of involvement)? Is there a problem with attrition (before participants complete program services)?
  12. What do you believe to be the major impacts/effects of the program on participants? (Indicate the outcomes and the levels typically achieved)

Performance Measurement and Information Flow:

  1. Please provide us with copies of all data collection requirements and forms for grantees. If you have reports from previous years, please provide us with copies. What data do you require beyond the GPRA outcomes, and why do you require it?
  2. Explain all the uses you make of data reported by grantees, including rewards, sanctions, use in future funding, completing GPRA reports, etc.
  3. Do you tie the evaluation data in any way to the GPRA data in assessing the program?
  4. Has the federal office implemented a standardized automated data system across states and program grantees to collect performance data? [If yes, obtain documentation on the data system, such as a copy of participant forms.] Alternatively, have grantees developed their own automated data systems? If yes, please provide a brief description of the various types of systems in use (e.g., is there great diversity in the types of systems in use)?
  5. How satisfied are program administrators with the existing performance monitoring and/or data system? Are there ways in which the performance monitoring or data systems might be improved? Is there any planned or ongoing effort to change either the types of performance information collected or the way in which these data are collected?
  6. What types of information would your agency ideally want to be able to measure regarding program performance (e.g., process and outcome measures)? To what extent would it be possible for states/grantees to report on these measures?
  7. As part of this study, we are exploring possible methodologies for identifying chronically homeless individuals. Our aim will be to develop a brief set of questions, characteristics, proxy measures, or indices that could be readily and efficiently determined at intake/enrollment in a program and which would assist in identifying a chronically homeless client. Do you have any suggestions of possible questions/characteristics that should be included in this index?
  8. Because this program may be one of several serving the participants, either concurrently or sequentially, do you think the performance measures used or proposed can isolate the effects of this program? Do you have any suggestions for dealing with this?

Discussion Guide for Initial Visits to
DHHS Homeless-Serving Programs:
PATH

Interviewee:
Date of Interview:

Background on the Program and Services 

  1. Please provide a brief history of the programs origins and evolution.
  2. What is the most recent annual budget for the program? What are the budget requests for the next several out years? Is all the money allocated by formula? If other distribution means are used, please describe.
  3. Please describe the general structure of the program:
    1. What is the role of the federal government?
    2. What is the role for states?
    3. Who are the grantees? What is the distribution among states, cities and counties, CBOs, and other organizations?
    4. How common is it for a grantee to provide all or most of the programs services as opposed to using subcontractors/sub-grantees? What is the nature and distribution of the subcontractors/sub-grantees?
  4. Please provide a copy of the GPRA plan and outcome measures for this program and address the following questions:
    1. Are all the GPRA measures of equal importance to the agency? If not, please rank them.
    2. Are there any other performance measures beyond the GPRA measures that you currently use? If so, what are they, how are they tracked, and how do they rank compared to the GPRA measures?
  5. Are there any national, state, or local evaluations of the program? If so, would you characterize these evaluations as process studies, outcome studies, or net impact evaluations? How do you use the evaluations in measuring performance of your grantees? How do the evaluations tie in to your GPRA work? Can you provide us with several examples of evaluations, including some that are good and some that are poor in quality?
  6. How do the states distribute their funds? Do they sometimes run programs themselves? Do they sponsor competitions and/or do they distribute the funds to county and local government?
  7. How competitive is the program? What proportion of grant applications are funded? What differentiates those funded from those not funded?
  8. Once recruited, how do participants flow through the program (i.e., from recruitment, through intake and assessment, and into program services)?
  9. Please briefly describe the main types of services and activities participants receive (i.e., health care services; mental health services; substance abuse services; employment and training services; help resolving housing problems; other support services)  [note: indicate whether all participants receive a specific service or only some participants receive the service]?
  10. Do local programs typically partner with other agencies to provide services to participants (i.e., refer participants to other agencies for services)? If so, who are the other organizations and what services do they provide? How does the partnering work? Is there a subcontract, formal referral, informal referral, or some other approach?
  11. At what point in the process is an individual considered a program participant? Are program participants formally terminated from the program, and if so, when? How long do participants typically stay enrolled in the program (i.e., range/average length of involvement)? Is there a problem with attrition (before participants complete program services)?
  12. What do you believe to be the major impacts/effects of the program on participants? (Indicate the outcomes and the levels typically achieved)

Performance Measurement and Information Flow:

  1. Please provide us with copies of all data collection requirements and forms for grantees. If you have reports from previous years, please provide us with copies. What data do you require beyond the GPRA outcomes, and why do you require it?
  2. Explain all the uses you make of data reported by grantees, including rewards, sanctions, use in future funding, completing GPRA reports, etc.
  3. Do you tie the evaluation data in any way to the GPRA data in assessing the program?
  4. Has the federal office implemented a standardized automated data system across states and program grantees to collect performance data? [If yes, obtain documentation on the data system, such as a copy of participant forms.] Alternatively, have grantees developed their own automated data systems? If yes, please provide a brief description of the various types of systems in use (e.g., is there great diversity in the types of systems in use)?
  5. How satisfied are program administrators with the existing performance monitoring and/or data system? Are there ways in which the performance monitoring or data systems might be improved? Is there any planned or ongoing effort to change either the types of performance information collected or the way in which these data are collected?
  6. What types of information would your agency ideally want to be able to measure regarding program performance (e.g., process and outcome measures)? To what extent would it be possible for states/grantees to report on these measures?
  7. As part of this study, we are exploring possible methodologies for identifying chronically homeless individuals. Our aim will be to develop a brief set of questions, characteristics, proxy measures, or indices that could be readily and efficiently determined at intake/enrollment in a program and which would assist in identifying a chronically homeless client. Do you have any suggestions of possible questions/characteristics that should be included in this index?
  8. Because this program may be one of several serving the participants, either concurrently or sequentially, do you think the performance measures used or proposed can isolate the effects of this program? Do you have any suggestions for dealing with this?

Discussion Guide for Initial Visits to
DHHS Homeless-Serving Programs:
Runaway and Homeless Youth Program

Interviewee:
Date of Interview:

Background on the Program and Services (Basic Center Program) 

  1. Please provide a brief history of the programs origins and evolution.
  2. What is the most recent annual budget for the program? What are the budget requests for the next several out years? Is all the money allocated by formula? If other distribution means are used, please describe.
  3. Please describe the general structure of the program:
    1. What is the role of the federal government?
    2. What is the role for states?
    3. Who are the grantees? What is the distribution among states, cities and counties, CBOs, and other organizations?
    4. How common is it for a grantee to provide all or most of the programs services as opposed to using subcontractors/sub-grantees? What is the nature and distribution of the subcontractors/sub-grantees?
  4. Please provide a copy of the GPRA plan and outcome measures for this program and address the following questions:
    1. Are all the GPRA measures of equal importance to the agency? If not, please rank them.
    2. Are there any other performance measures beyond the GPRA measures that you currently use? If so, what are they, how are they tracked, and how do they rank compared to the GPRA measures?
  5. Are there any national, state, or local evaluations of the program? If so, would you characterize these evaluations as process studies, outcome studies, or net impact evaluations? How do you use the evaluations in measuring performance of your grantees? How do the evaluations tie in to your GPRA work? Can you provide us with several examples of evaluations, including some that are good and some that are poor in quality?
  6. How do the states distribute their funds? Do they sometimes run programs themselves? Do they sponsor competitions and/or do they distribute the funds to county and local government?
  7. How competitive is the program? What proportion of grant applications are funded? What differentiates those funded from those not funded?
  8. Once recruited, how do participants flow through the program (i.e., from recruitment, through intake and assessment, and into program services)?
  9. Please briefly describe the main types of services and activities participants receive (i.e., health care services; mental health services; substance abuse services; employment and training services; help resolving housing problems; other support services)  [note: indicate whether all participants receive a specific service or only some participants receive the service]?
  10. Do local programs typically partner with other agencies to provide services to participants (i.e., refer participants to other agencies for services)? If so, who are the other organizations and what services do they provide? How does the partnering work? Is there a subcontract, formal referral, informal referral, or some other approach?
  11. At what point in the process is an individual considered a program participant? Are program participants formally terminated from the program, and if so, when? How long do participants typically stay enrolled in the program (i.e., range/average length of involvement)? Is there a problem with attrition (before participants complete program services)?
  12. What do you believe to be the major impacts/effects of the program on participants? (Indicate the outcomes and the levels typically achieved)

Performance Measurement and Information Flow:

  1. Please provide us with copies of all data collection requirements and forms for grantees. If you have reports from previous years, please provide us with copies. What data do you require beyond the GPRA outcomes, and why do you require it?
  2. Explain all the uses you make of data reported by grantees, including rewards, sanctions, use in future funding, completing GPRA reports, etc.
  3. Do you tie the evaluation data in any way to the GPRA data in assessing the program?
  4. Has the federal office implemented a standardized automated data system across states and program grantees to collect performance data? [If yes, obtain documentation on the data system, such as a copy of participant forms.] Alternatively, have grantees developed their own automated data systems? If yes, please provide a brief description of the various types of systems in use (e.g., is there great diversity in the types of systems in use)?
  5. How satisfied are program administrators with the existing performance monitoring and/or data system? Are there ways in which the performance monitoring or data systems might be improved? Is there any planned or ongoing effort to change either the types of performance information collected or the way in which these data are collected?
  6. What types of information would your agency ideally want to be able to measure regarding program performance (e.g., process and outcome measures)? To what extent would it be possible for states/grantees to report on these measures?
  7. As part of this study, we are exploring possible methodologies for identifying chronically homeless individuals. Our aim will be to develop a brief set of questions, characteristics, proxy measures, or indices that could be readily and efficiently determined at intake/enrollment in a program and which would assist in identifying a chronically homeless client. Do you have any suggestions of possible questions/characteristics that should be included in this index?
  8. Because this program may be one of several serving the participants, either concurrently or sequentially, do you think the performance measures used or proposed can isolate the effects of this program? Do you have any suggestions for dealing with this?

View full report

Preview
Download

"report.pdf" (pdf, 1.01Mb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®