Feasibility Study for the
Evaluation of DHHS Programs
That Are or May Be Operated
Under Tribal Self-Governance

ACKNOWLEDGEMENTS

Cynthia Helba and Kathy Langwell, Westat, and Jo Ann Kauffman, Kauffman and Associates, Inc., prepared this Final Report.  W. Sherman Edwards and Brian Colhoff, Westat, Victor Paternoster, Kauffman and Associates, Inc., Janet Sutton, Project HOPE Center for Health Affairs, and Frank Ryan, project consultant, made substantial contributions to the study.  We would like to thank the members of the Technical Working Group for their assistance and thorough reviews of draft reports throughout the study.  In addition, we are grateful to the Tribes/Tribal organizations that volunteered to participate in site visits that were conducted for the study, including:  Bois Forte Band of Chippewa, Choctaw Nation, Hopi, Little Traverse Bay Band of Odawa Indians, Port Gamble S’Klallam Tribe, and the Yukon-Kuskokwim Health Corporation.  Several individuals at the Indian Health Service also contributed generously of their time and provided assistance with data review activities.  Finally, we would like to thank Andy Rock and Tom Hertz, OASPE/DHHS Task Order Managers for the project, who provided valuable guidance and insights throughout the project.

DISCLAIMER

The conclusions and discussions presented in this report are those of the authors and do not necessarily reflect the views of the Office of the Assistant Secretary for Planning and Evaluation, the Indian Health Service, other agencies within the U.S. Department of Health and Human Services, or of the members of the Technical Working Group that provided advice and assistance to the study team.

TABLE OF CONTENTS

Chapter

1. SUMMARY AND FINDINGS, BACKGROUND, AND OBJECTIVES
   1. Summary and Findings
   2. Background of the Study
   3. Objectives of the Study
   4. Organization of this Draft Final Report
2. ISSUES AND CONSIDERATIONS FOR EVALUATION FEASIBILITY
   1. Overview
   2. Perspectives on Evaluation Research:  Congress, DHHS, and Tribes
   3. Feasibility Considerations
3. STUDY METHODOLOGY
   1. Approach to Assessing Feasibility of an Evaluation
   2. Overview of Background Information Collection
   3. Methods for Obtaining Input from Technical Working Group and Others
   4. Methodology for Site Visits and Tribal Data Review
   5. Methodology for Small Group Discussions
4. SITE VISIT FINDINGS
   1. Overview
   2. Availability, Accessibility, and Quality of Data on History of Self-Governance and Program Management
   3. Availability, Accessibility, and Quality of Data on DHHS Programs That
      May Be Operated by Tribes Under A Self-Governance Demonstration
   4. Summary of Availability of Data
5. DISCUSSION GROUP FINDINGS
6. EVALUATION FEASIBILITY
   1. Overview and Approach
   2. Tribal Consultation in Development of Any Future Evaluation
   3. Description of Three Illustrative Evaluation Models for Assessment of the  Feasibility of Evaluating DHHS Programs that May Be Operated by Tribes Under a Self-Governance Demonstration
   4. Comprehensive Evaluation Model – DHHS Programs That May Be Operated By Tribes Under a New Self-Governance Demonstration
   5. Limited Evaluation Model
   6. Evaluation Model Using Aggregate Reporting Data
   7. Comparison of Feasibility of the Three Illustrative Models

1. Technical Working Group Members
2. Site Visit Data Collection Forms
3. Matrix of Issues and Concerns Raised at Discussion Groups  at NIHB and DHHS/DOI Self-Governance Conferences
4. Description of Illustrative Evaluation Models
5. Cost Estimates for Each Illustrative Model
6. Feasibility of Evaluating Indian Health Service Programs Operated by Tribes Under Self-Governance
7. Detailed Description Of Illustrative Comprehensive Evaluation Model: Indian Health Service Programs Operated By Tribes Under Self-Governance

1. SUMMARY AND FINDINGS, BACKGROUND, AND OBJECTIVES

1.1 - Summary and Findings

1.1.1 - Purpose of Study

The Evaluation Feasibility Study was designed to provide information to the Office of the Assistant Secretary, U.S. Department of Health and Human Service (OASP) and the Tribes on several questions:

• Would an evaluation of Tribal Self-Governance be feasible?
• What would be the important evaluation issues and research questions?
• How would these issues and questions be addressed?
• Would there be sufficient data available to permit an evaluation to be conducted?
• What would be the costs of each alternative approach?

The Evaluation Feasibility Study was structured to obtain input and views of Tribal representatives, through a Technical Working Group, site visits, and discussion groups held at national American Indian and Alaska Native (AI/AN) conferences.  Technical Working Group members and other Tribal representatives provided input and comments on a range of issues, including feasibility considerations, the type of evaluation activities that would be desirable, and the potential benefits that evaluation could provide to Tribes. 

It is important to note that, while the purpose of this project has been to provide information helpful to the design of an evaluation, the project was not designed to produce a definitive evaluation design and methodology.  Results of this Study identify and describe feasible options for an evaluation and provide observations helpful to efforts on the part of DHHS and the Tribes regarding the range of issues that should be addressed.

1.1.2 - Findings: Tribal Perspectives

The Technical Working Group and other representatives of self-governing Tribes who provided guidance and input to this project emphasized a number of points that were important to the development of the study findings:

1. Many Tribes that are involved in Self-Governance^[1] are convinced that Self-Governance is a successful and effective mechanism for providing services to Tribal members.  Self-Governance can ensure that services are available to Tribal members and permits programs to be designed to meet specific Tribal priorities and circumstances.
2. Many Self-Governance Tribes understand and support the concept of evaluation and recognize the benefits that evaluation processes and findings may provide to Tribes.  These benefits may include improved data systems/data availability, evidence of the effectiveness and success of Self-Governance, and information on “best practices” for management.
3. The design of any potential evaluation should be developed through consultation between DHHS and the Tribes.  It is important that DHHS provide for the costs of any evaluation’s data and reporting burden on Tribes, if these costs are significant.  It would also be important that there be clearly defined limits on the timeframe within which evaluation-related data would be submitted by Tribes, if the data needed for evaluation was more extensive than the usual Self-Governance program reporting requirements.  In addition, Tribes are concerned about the total costs to DHHS of an evaluation and want to be certain that these total costs are justified by the potential benefits that might be obtained from an evaluation.
4. Any evaluation design should take into account the goals and principles of Self-Governance.  Self-Governance offers Tribes the opportunity for flexibility to develop and re-structure programs to meet specific Tribal objectives.  Any evaluation design should be similarly flexible in defining the outcomes that would be measured.
5. The design of any potential evaluation should also consider the fact that Tribally-managed programs are under-funded, generally.  This is a particularly important consideration if external comparison groups/strategies were to be used in an evaluation, since joint Federal-State funding of some programs is considerably higher than the funding provided to Tribes for those programs.
6. Evaluation of existing DHHS/Indian Health Service (IHS) Self-Governance programs that have been operating for over a decade would be different from an evaluation of a potential new DHHS Self-Governance demonstration program.  The Indian Health Service has data that could be used for such an evaluation, if DHHS chose to conduct an evaluation. Technical Working Group members stressed, however, that an evaluation of DHHS programs that may be operated under a potential new demonstration would be more valuable and provide more useful information to DHHS and to Tribes. (A discussion of the feasibility of evaluating Indian Health Service programs operated under Self-Governance compacts is presented separately in Appendix F to this Report.)

1.1.3 - Findings: Evaluation Feasibility

The methodology for assessing the feasibility of evaluating DHHS programs that in the future may be operated by Tribes under Self-Governance included:  1) establishment of a Technical Working Group that provided guidance and input throughout the project; 2) extensive background review of current DHHS programs operated by Tribes under contracts and IHS and Bureau of Indian Affairs (BIA) contracts and compacts, including a review of the literature to identify and assess existing evaluations of Self-Governance programs and discussions with DHHS program staff to identify current reporting requirements; 3) site visits to five Tribes and one Tribal organization ^[2] to obtain information about data that are currently available to or produced regarding DHHS programs operated by the Tribes and the availability of staff who could provide background and history of the Tribal  management of DHHS programs; and 4) discussion groups with a larger number of Tribal representatives to obtain feedback and additional information.

Three alternative illustrative evaluation models were identified:  1) a Comprehensive evaluation model with qualitative and quantitative components and extensive data needs; 2) a Limited evaluation model, focused on a smaller set of key evaluation issues and with less extensive data needs; and 3) an Aggregate evaluation model, that would use routine program reports to examine processes, objectives, and outcomes of DHHS programs that may be managed by Tribes under Self-Governance.  These three models were then assessed for feasibility, based on the background information, site visit findings, discussion group findings, and input from the Technical Working Group.  Feasibility considerations included:

• Availability of appropriate comparison groups or strategies.
• Availability of existing data and/or the feasibility of collecting additional data needed for evaluation.
• Total costs of the evaluation approach and the trade-offs between costs and the potential value of evaluation findings.
• Tribal support for the evaluation approach, based on considerations that include potential costs to DHHS and participating Tribes and potential benefits that might be obtained.

Results of the feasibility assessment included:

• All three of the evaluation models were determined to be technically feasible. That is, necessary data either are available or could be collected and appropriate comparison strategies could be constructed.
• Costs would be quite high for the Comprehensive evaluation model for evaluation of DHHS programs under a potential new Self-Governance demonstration, but results would be rigorous and could provide a wide range of potentially useful information.
• The Limited evaluation model is technically feasible and less costly than the Comprehensive evaluation model. It would likely produce useful results regarding the evaluation issues addressed.
• The Aggregate evaluation model, using standard reporting data, is technically feasible and least costly. It would produce some useful findings, but less detailed information than the Comprehensive or Limited evaluation models.

The Technical Working Group reviewed the evaluation feasibility study findings and, as a group, agreed that each of the three evaluation models was feasible and might produce useful findings for the Tribes and for DHHS.  The estimated costs associated with the Comprehensive evaluation model were considered high, relative to the potential value of the findings that would be generated.  There was general agreement within the Technical Working Group that the Limited evaluation model or the Aggregate evaluation model would be preferred to the Comprehensive evaluation model, since either of these alternatives could produce useful results at a more modest cost.

The Technical Working Group also emphasized that many Tribes would be supportive of and willing to participate in an evaluation of DHHS programs that may be operated by Tribes under a new Self-Governance demonstration.  The design of such a demonstration should result from consultation between DHHS and Tribes on the specific issues to be addressed and data that would be needed.  TWG members also stated that, because it would be important for DHHS to provide financial support for the development of data systems needed for any evaluation, DHHS should consider those costs as part of the total costs of any evaluation that might be designed.   Consideration of whether such consultation and agreement would need to precede or could equally well follow the enactment of any possible demonstration authority by Congress is beyond the scope of this report.

Finally, an important feasibility issue is the timing of an agreement between DHHS and the participating Tribes on the evaluation issues that would be addressed and on the data that would be provided by participating Tribes for an evaluation.  This agreement would be essential prior to the implementation of a new demonstration program to assist participating Tribes to be aware of data requirements and to establish procedures to collect data from the initiation of a demonstration. In the absence of such an agreement prior to implementation of a demonstration program, it would be difficult to ensure that appropriate and necessary data would be available for an evaluation.

1.2 - Background of the Study

In the Tribal Self-Governance Amendments of 2000 (P.L. 106-260), Congress re-affirmed its commitment to Tribal Self-Governance.  In the Preamble to the Act, the Congress defined the goal of Self-Governance as “to permit an orderly transition from Federal domination of programs and services to provide Indian Tribes with meaningful authority, control, funding, and discretion to plan, conduct, redesign, and administer programs, services, functions, and activities (or portions thereof) that meet the needs of individual Tribal communities.”

The Act established Tribal Self-Governance of Indian Health Service programs on a permanent basis.  In addition, the Congress directed the Secretary of DHHS to “conduct a study to determine the feasibility of a Tribal Self-Governance demonstration project for appropriate programs, services, functions, and activity (or portions thereof) of the agency [HHS].”  The Office of the Assistant Secretary for Planning and Evaluation conducted the Tribal Self-Governance Demonstration Feasibility Study in 2001-2002.  The Final Report on the Study, submitted to Congress in March 2003, identified 11 DHHS programs as “feasible for inclusion in a Tribal Self-Governance demonstration project”. These 11 programs^[3] are: 

• Administration on Aging
1. Grants for Native Americans
• Administration for Children and Families
  2. Tribal Temporary Assistance for Needy Families
  3. Low Income Home Energy Assistance
  4. Community Services Block Grant
  5. Child Care and Development Fund
  6. Native Employment Works
  7. Head Start
  8. Child Welfare Services
  9. Promoting Safe and Stable Families
  10. Family Violence Prevention:  Grants for Battered Women’s Shelters
• Substance Abuse and Mental Health Services Administration
  11. Targeted Capacity Expansion

There are Tribes currently managing each of these DHHS programs that were determined feasible for inclusion in a Tribal Self-Governance Demonstration project, under contractual arrangements or grant awards.^[4]  A Self-Governance Demonstration program could, as detailed in the Final Report, permit a simpler, multiple-program application process and simpler and consolidated reporting requirements.  Importantly, the Demonstration program could provide “Tribes with the flexibility to change programs and reallocate funds among programs” to better address specific Tribal community priorities.

Congressional action is necessary to authorize a DHHS Tribal Self-Governance Demonstration, if such a demonstration were enacted.  The study reported here examines how an evaluation of outcomes and successes of DHHS programs managed by Tribes under Self-Governance might be conducted. While a number of assessments of Tribally-managed programs have been conducted, these have been primarily qualitative in nature. The Office of the Assistant Secretary for Planning and Evaluation (OASPE) was interested in determining the feasibility of conducting an evaluation that could include both qualitative and quantitative analyses of processes and outcomes associated with DHHS programs managed by Tribes under Self-Governance.

In September 2002, DHHS contracted with Westat, and its subcontractor, Kauffman and Associates, Inc., to conduct a study that would provide background information and assess the feasibility of conducting a rigorous and defensible evaluation of DHHS programs managed by Tribes under Self-Governance.

1.3 - Objectives of the Study

The Evaluation Feasibility Study was designed to provide information to DHHS and to Tribes on several questions:

• Would an evaluation of DHHS programs operated under a new Tribal Self-Governance demonstration program be feasible?
• What evaluation issues and research questions would be important to address in an evaluation?
• What measures – both qualitative and quantitative – would be appropriate to use to address each of the selected evaluation issues and research questions?
• Would there be data available, within DHHS and from Tribes managing DHHS programs, which would permit an evaluation to be conducted, using a rigorous methodology that would be likely to produce reliable results?
• What would be the cost implications of alternative feasible evaluation strategies?

This project was intended to provide information helpful to the design of an evaluation and was not designed to produce a definitive evaluation design and methodology. Results of this Study were intended only to provide information on feasible options for an evaluation and considerations that can be used by DHHS in consultation with the Tribes about an evaluation that could be conducted and the range of issues that could be addressed.

1.4 - Organization of the Final Report

This Final Report on the Evaluation Feasibility Study provides background information and assesses the feasibility of conducting an evaluation of DHHS programs that may be managed by Tribes under a potential new Self-Governance demonstration.  Throughout the 15-month project, guidance and input were provided to the study team and to OASPE by the project’s Technical Working Group (TWG).^[5] In addition, preliminary findings of the site visits and data reviews were presented at discussion sessions held at three national conferences: National Indian Health Board, National DHHS/DOI Tribal Self-Governance meetings, and National Congress of American Indians.  The issues raised and perspectives of the TWG and participants in the discussion groups are reflected in the findings presented throughout this Final Report. 

Section 2 of this Report provides an overview of considerations for the feasibility of conducting an evaluation of DHHS programs managed by Tribes under Self-Governance and describes key feasibility issues that guide the study. The study methodology and background information activities conducted are described in Section 3.  Findings from the site visits and data reviews are presented in Section 4 and results of the discussion groups held at national conferences are presented in Section 5.  In Section 6, three illustrative evaluation models are described and issues and considerations for the feasibility of conducting these alternative evaluation models are discussed. 

2. ISSUES AND CONSIDERATIONS FOR EVALUATION FEASIBILITY

2.1 - Overview

The Evaluation Feasibility Study was initiated to provide information to the Department of Health and Human Services on the potential to evaluate DHHS programs that may be managed by Tribes under Self-Governance, if a demonstration program were to be authorized by Congress.^[6] Most demonstration programs within the Department frequently are developed with an evaluation component that is designed to assess the program’s operations, processes, and outcomes. 

The Department of Health and Human Services determined that examining the potential for evaluating DHHS programs that may be managed by Tribes under a potential new Self-Governance demonstration would be useful.  The focus of this planning study was to determine whether it would be feasible to conduct an evaluation that included quantitative measurement of process and outcomes, as well as qualitative information on program objectives, implementation, and operations.

Early in the project, it became clear that designing an evaluation of DHHS programs operated under Tribal Self-Governance was a more complex task than is the norm for other DHHS programs.  Tribal Self-Governance is intended to allow Tribes to manage their own programs with flexibility and with minimal requirements for reporting to Federal agencies.  The most common approach to evaluation requires substantial data collection that is consistent and comparable across organizations participating in a demonstration and a set of common objectives/outcome measures that may be the focus of the evaluation.  The goal of flexibility that is a principle of Self-Governance offers challenges in determining a consistent, comparable set of outcome measures and in defining an appropriate methodology for measuring outcomes across participating Tribes. The nature of the government-to-government relationship between the Federal government and individual Tribes also requires consultation and agreement on the type and the extent of any evaluation program. The Technical Working Group (TWG) stressed the importance of these issues at the initial meeting with the project team in February 2003.  In addition, the TWG members also stressed that most Self-Governance Tribes are convinced that Self-Governance is a successful and effective means for delivering services to Tribal members. 

Generally, many Tribes appear to be supportive of and willing to participate in any potential evaluation of DHHS programs that may be operated by Tribes under a new Self-Governance demonstration.  Guidance provided by the TWG for assessing the feasibility of such an evaluation emphasized that there are a number of issues, however, that needed to be addressed in determining the feasibility of any particular evaluation model. 

First, the TWG stated it is important that the design of any evaluation be clear in communicating the objectives and benefits of the evaluation to Tribes and provide assurance that evaluation would not have detrimental effects on the future of Self-Governance.  The TWG also recommended that design of any evaluation should take into consideration that the underlying goal of Self-Governance is to offer Tribes flexibility to structure programs to better meet local priorities and, consequently, any evaluation that examines a set of outcomes solely Federally-determined would be inappropriate.  Instead, there should be an emphasis on flexibility in setting outcomes to be measured based on Tribal priorities and objectives.

The TWG also noted that no evaluation method should require Self-Governance Tribes to provide more extensive reporting than that required of Tribes carrying out programs under current legal authority.  Nor should any evaluation method impose extensive and burdensome data reporting on Tribes, rather than the minimal reporting that is one of the principles of Self-Governance.  If the burden on Tribes, and associated costs of data collection, are more than nominal, then an evaluation design should include funding to support the costs.  In addition, if a specific evaluation design includes extensive data collection, it is essential that there be clear and specific limitations on the time period for the evaluation and a specified date upon which data collection and reporting related to the evaluation would end.

Finally, members of the TWG noted that Self-Governance is an evolving process with new programs.  It may take two or three years for a Tribe managing a new program to identify priorities and develop more effective operational and management strategies.  Therefore, any potential evaluation should be designed to take into account the evolution of the program and the fact that Tribal goals and priorities may not be determined until the second or third year.

These issues and considerations guided the development of the evaluation feasibility study and the analysis of the feasibility of alternative evaluation approaches.

2.2 - Perspectives on Evaluation Research: Congress, DHHS, and Tribes

When Congress authorizes a demonstration program, it is generally with an understanding that it will be designed and implemented for a limited period of time to determine whether the new approach can effectively and efficiently meet specific goals.  Evaluation of demonstration programs provides information and evidence on the process through which the programs are implemented, operational issues, and impacts and outcomes of the demonstration programs, relative to the goals of the programs. Evaluation findings may also provide information that can be used to refine and improve the demonstration program as it transitions to permanent status.  Because evaluation is an accepted tool for assessing new programs, Congress often requires that an evaluation be conducted of new demonstration programs that it authorizes.

Agencies within the Department of Health and Human Services may have an interest in evaluating new strategies for delivering services to target populations. Management and program staff who have responsibility for specific programs have often worked in their fields for many years to develop program designs they believe are effective to provide services and meet defined needs of the population they serve.  When a new delivery strategy is undertaken, there may be concerns about whether the program objectives will continue to be achieved and whether the target population will be as well served as it was when DHHS dictates the strategy for program management.  Evaluations of how the newly structured delivery systems operate and meet the needs of the target population may allay concerns and provide increased support for the new delivery model. 

Tribes may perceive evaluation as having the potential to provide support for and evidence regarding the program improvements that Tribes can achieve through Self-Governance.  Evaluation may also provide useful information on innovative management approaches and on “best practices” that could be adopted by more Tribes and other recipients of Federal program funds. In addition, evaluation findings could provide evidence that suggests that additional funding of programs would be warranted in order for Tribes to achieve specific objectives and desirable outcomes.

2.2.1 - Potential Benefits of an Evaluation

If an evaluation of DHHS programs that may be operated by Tribes under a new Self-Governance demonstration were to be conducted, it may have several benefits both to DHHS and to Tribes.  DHHS program managers and staff which have had responsibility for administering programs that the Tribes would manage under a demonstration would perceive an evaluation as consistent with the normal approach to assessing the effectiveness and success of any new program arrangements undertaken within the Department.  Evaluation could provide evidence that the needs of clients are being met under the new management structure, even though some aspects of the program may be different than under direct Federal program management.  In addition, results of an evaluation could provide information that would increase understanding of Tribal issues and goals among DHHS program managers and staff and the benefits and successes of Tribal Self-Governance in better meeting the unique needs of Tribal members.

Evaluation of DHHS programs managed under Tribal Self-Governance could provide useful information to Tribes, as well.  Results could provide information on “best practices” and innovative programs that could be used by other Tribes to improve services and performance in program management.  There is also the potential for the findings to demonstrate that Tribal Self-Governance is an effective method for improving services and meeting needs of individual Tribes and Tribal members that would provide support for further expansion of Self-Governance to additional DHHS and other agencies’ programs.

2.3 - Feasibility Considerations

The feasibility of conducting an evaluation would be dependent on a number of issues and considerations.  These include:

• Appropriate Comparison Groups.  Agreement by DHHS and Tribes on appropriate and acceptable – to DHHS and to Tribes – comparison groups are critical to the feasibility of any potential evaluation.  Evaluation research involves comparing a new program’s operations and impacts relative to what would have been observed in the absence of the new program.  Pre-post comparisons are generally accepted approaches, but do not take into account underlying trends and changes that may affect what is observed in the new program.  External comparison groups are usually defined and examined to adjust pre-post data for any outside trends that may affect programs. 
• Data Availability.  The availability of data for the pre-post demonstration period is a necessary condition for conducting an evaluation of DHHS programs operated by Tribes under a Self-Governance demonstration.  Similarly, if external comparison groups are to be used, comparable and consistent data must be available for the relevant time periods of the evaluation.
• Costs to DHHS and to Participating Tribes.  Costs of any evaluation approach considered are an important consideration in assessing evaluation feasibility.  Some evaluation alternatives may involve much higher costs than others that might be prohibitive.  Trade-offs may be considered between the comprehensiveness and rigor of evaluation alternatives and costs and the potential value of the findings that may be produced.
• Tribal Support for an Evaluation.  The likelihood that Tribes will support an evaluation will be affected by the goals of the evaluation, the nature and extent of consultation between the Tribes and DHHS on the goals and processes of the evaluation, and the costs and burden of participation to the Tribes. In addition, Tribal support may also depend on the total costs of an evaluation, even if these costs are fully borne by DHHS, relative to the potential value and benefits that may be obtained from the evaluation.

An additional important feasibility issue is the timing of an agreement between DHHS and the participating Tribes on the evaluation issues to be addressed and on the data that will be provided by participating Tribes for the evaluation.  This agreement would be essential prior to the implementation of the new demonstration program to assist participating Tribes to be aware of data requirements and to establish procedures to collect data from the initiation of the demonstration. In the absence of agreement prior to the demonstration and specification of necessary data, it would be difficult to ensure that appropriate and necessary data would be available for the evaluation

3. STUDY METHODOLOGY

3.1 - Approach to Assessing Feasibility of an Evaluation

The Evaluation Feasibility Study involved the following activities:

• Establishment of a Technical Working Group that provided ongoing input and review of interim study products, as well as guidance on study objectives and processes;
• Broad communication about the project to all Tribal leaders, through mailings, a DHHS website (http://aspe.hhs.gov/SelfGovernance/Evaluation/actions.htm), and organization of discussion groups at national conferences:
• Extensive background information collection and summaries of findings;
• Review of DHHS program data and reporting requirements; and
• Conduct of site visits to six Tribes to collect background information and review data availability that might support an evaluation, if one were to occur.

3.2 - Overview of Background Information Collection

During the initial months of this project, a substantial amount of information was assembled by the project team, as background for understanding and laying the groundwork for  the Evaluation Feasibility Study.  Each of these activities and the associated reports that were prepared are described in this section.^[7]

3.2.1 - Review of Previous Research on Tribal Self-Governance/Tribal Management of Federal Programs

The Literature Review was conducted to provide a foundation for the development of the evaluation issues and related data requirements to guide the design of the feasibility study.  The objectives of the literature review included:

• Identification of existing studies and evaluations of Tribal Self-Governance and/or Tribal management of health and social service programs;
• Identification of data limitations and other factors that pose barriers to conducting comprehensive evaluations of Self-Governance and Tribal management of health and social service programs.
• Review of the methodologies and data sources used in previous studies, in order to assess both analytic rigor and the extent to which the findings are of general application;
• Synthesis of  available evidence and findings from existing studies; and
• Assessment of the implications of these findings for the Tribal Self-Governance Evaluation Feasibility Study.

Tribal Self-Governance of Indian Health Service programs have been in place for a decade and extensive detailed data are submitted by Self-Governance Tribes to IHS.  These data would make it possible to conduct a quantitative evaluation of Indian Health Service programs managed by Tribes under Self-Governance.  The evaluations of Self-Governance of IHS programs that have been conducted to date, however, have been primarily qualitative. There has been limited evaluation of other programs managed by Tribes under either Self-Governance or under contracts and grants. 

Most of the studies identified that have evaluated Self-Governance or Tribal management of health or social programs employ qualitative techniques, such as key informant interviews, which rely on stakeholders’ perceptions to reach conclusions about program effectiveness.  These qualitative studies provide insight into how various Tribes structure their health and social service programs, the characteristics of Tribal residents participating in these programs, and successes encountered in program implementation.  Further, these studies – particularly those that focused on the Tribal Temporary Assistance to Needy Families program – effectively highlight how social and economic conditions on Reservations, such as the high rate of poverty, high unemployment rates, and the lack of an economic base, may pose substantial barriers to achieving the intended goals of these programs.  These studies do not, however, provide definitive quantitative evidence on the extent to which and how Tribally-managed health and social service programs have operated to better meet the needs of their members. Most studies do not incorporate a comparison group in their design and, therefore, it is not possible to determine how persons participating in Tribal programs fare compared to how they would have fared if control over these programs were still vested with the Federal or State government.

Previous research on process, structure, and impacts of Tribal management of health and social services is limited in major ways:  1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not useful to other Tribes or other Federal grantees; and 3) adequate and comparable data across Tribally-managed programs and between Tribally-managed programs and Federal and State managed programs are not available.

Given these findings, a primary focus of the Evaluation Feasibility Study was to review and identify potential sources of data that would be adequate to permit a quantitative evaluation of relevant issues. 

3.2.2 - Legislative History and Development of Tribal Self-Governance

In addition to the literature review, a summary of the legislative history and development of Tribal Self-Governance was prepared to provide background for understanding the context within which Self-Governance has evolved and the underlying principles on which the development of Tribal Self-Governance has been based.  The authority for Tribal management of Federally-funded programs was initially provided by Congress under the Indian Self-Determination and Education Assistance Act of 1975 (P.L. 93-638).^[8] Title I of that Act authorized Tribes to assume management of Bureau of Indian Affairs/DOI and Indian Health Service programs through contractual agreements.  From 1975 to the present, Congress has expanded the opportunities for Tribes to manage their own programs and has increased the degree of Tribal authority and discretion in management.  P.L. 100-472 amended P.L. 93-638 in 1988 to add Title III, which authorized the Tribal Self-Governance Demonstration Project that allowed Tribes to assume greater control over BIA programs that they managed, including consolidation and re-design of programs to better meet individual Tribal priorities and needs.

In 1992, as part of P.L. 102-477, Congress extended the Title III self-governance demonstration to provide for Tribal self-governance of Indian Health Service programs.  In that same Act, Congress created a program that permitted Tribes to consolidate employment and work-related programs from four Federal agencies.   Based on the perceived success of these demonstration projects, notwithstanding the lack of quantitative evaluations noted in this Report, Congress made Tribal self-governance authority permanent within BIA in 1994 (Title IV), and made permanent Tribal self-governance of IHS programs in 2000 (Title V).

Tribes manage other DHHS health and social service programs, under competitive contracts and formula grants.  These Tribally-managed programs do not generally offer the flexibility of program design and use of funds that self-governance provides to Tribes, and often require extensive program-specific application processes and detailed separate reporting requirements.

3.2.3 - Tribal Matrix of Programs Managed Under Contracts and Compacts

As background for the Evaluation Feasibility Study, OASPE was interested in determining the extent to which Tribes are currently managing DHHS or other Federally-funded programs under compacts, contracts, and grants.  These other programs include programs of the Department of the Interior, Bureau of Indian Affairs carried out under the “477” program (P.L. 102-477).^[9]  The information on Tribes that are currently managing programs also provided background information for recruiting six Tribes to participate in the site visit component of the Evaluation Feasibility Study.

The construction of the Tribal Matrix and identification of programs that are managed by each Tribe required: 1) identification of each Federally-recognized Tribe (including those Tribes in Alaska and other States that have authorized a Tribal organization to carry out programs on their behalf); and 2) identification of data sources and individuals in the Federal government that could provide information on Tribal management of the specific DHHS and BIA programs of interest.

A complete list of all Federally-recognized Tribes was obtained from the Federal Register^[10].  This list was then cross-referenced with Indian Health Service information to match Tribes in Alaska and other States to the Tribal organizations they have authorized to carry out programs on their behalf.

Data were obtained from BIA staff on Tribal management of BIA programs under Title I self-determination contracts and under Title IV Self-Governance compacts, as well as Self-Governance under P.L. 102- 477 provisions. Project staff searched each DHHS program area web site, as an initial step, to determine whether the program maintained a list of Tribes and Tribal organizations that hold contracts, grants, or compacts to manage specific programs.  Then, direct telephone contacts were made with DHHS staff associated with each program area to verify the accuracy of information obtained from the web site or to request information on Tribal management of programs. For five programs, the information required was maintained on the Federal agencies’ web sites; information was provided by program staff for the remaining seven DHHS programs.

3.2.4 - Tribal Population Characteristics and Related Data

To provide additional background information for the study, data were compiled from relevant sources that provided Tribal/Tribal organization-specific information on population size, age and gender distribution, socioeconomic characteristics, and any other variables that might be useful for describing and comparing Tribes.  This data compilation was intended to provide information for selecting and describing Tribes for the site visit component of the study, as well as information that could be useful to OASPE if an evaluation was conducted at some future time.

The Data Report provides information on a range of demographic and socioeconomic data that are useful for characterizing Tribes that manage both health and social service programs. Because of this broad interest, the Data Report is a compilation of information that may be relevant to Tribal management of many programs. 

Data sources that were used to develop the population and other characteristics, by Tribe, included the 2000 Census and documents from the Bureau of Indian Affairs and the Indian Health Service. There are serious limitations of these data sources and no good solutions are available to ensure that complete, accurate, and comparable data can be assembled for each Federally-recognized Tribe.  Despite these limitations, the Data Report provides some useful information on socio-economic and demographic characteristics of specific Tribes.

3.2.5 - Review of Reporting Requirements for  DHHS Programs That May Be Operated by Tribes Under a Self-Governance Demonstration

Current reporting requirements for each of the DHHS programs that have been identified as feasible to include in a potential new Self-Governance demonstration were also investigated by talking with several DHHS and Tribal program staff and reviewing the documents provided to us by the Tribes visited.  There is considerable variation in the level of detailed reporting that is required by different DHHS programs.  Some programs (e.g. Head Start) require lengthy and very detailed reporting on a large number of programmatic and outcomes measures.  Other programs do not have specific reporting requirements beyond minimal financial reports.  In addition to formal reporting requirements, some programs provide monitoring through ongoing contact between program staff and contracting Tribes. The current reporting requirements are described below.^[11]

Tribal Temporary Assistance for Needy Families (TTANF).     Currently, Tribes managing this program are required to submit (electronic submission is preferred) family-level and individual-level data elements for families receiving TTANF. For the family, these data elements include funding stream, number of family members, type of family for work participation, receiving subsidized housing, receiving medical assistance, receiving food stamps and amount, receiving subsidized child care and amount, child support, and family cash resources.  At the individual level, Tribes are required to submit characteristics such as adult and minor child head-of-household characteristic such as date for birth, ethnicity, gender, receipt of disability benefits, marital status, relationship to head of household, parent with minor child in the family, needs of pregnant women, educational level, citizenship, cooperation with child support, employment status, and work participation status.  The child characteristics submitted by TTANF grantees include family affiliation, race/ethnicity, gender, receiving disability benefits, relationship to head of household, educational level, amount of unearned income,

Low Income Home Energy Assistance Program.     Currently, Tribes managing this program are required to provide the Household Service Report—Short Format or a letter containing similar information.  This information includes number of households receiving the following types of assistance: heating, cooling, winter/year round crisis, summer crisis, or weatherization.

Community Services Block Grant.     Currently, there is no specific Federal required reporting form that documents services or activities conducted for this program.

Child Care and Development Fund.     The standard Child Care and Development Fund annual aggregate program report (ACF-700 form) requires the following information: number of families and children receiving services, age breakdown for children receiving services, reasons for needing childcare (e.g., working, in school), number of hours services provided, amount of CCDF subsidy, amount of parent co-payment, and poverty status of families receiving services.  A supplemental narrative report to the ACF-700 form requests general child care information, and is not restricted to CCDF-funded activities.  Financial reporting is required on the ACF-696-T form.

Native Employment Works.     Current Federal reporting requirements include a Program Report with a narrative section that compares achievements for the year to their plan for the year.  It also summarizes significant barriers to implementation, provides explanations for variances with the plan, and describes actions taken.  The Program Report also includes a statistical report that provides the following information: number of clients served characteristics of clients served (e.g., age, sex, TANF recipients), number of clients participating in types of NEW activities and services (e.g., classroom training, on-the-job training, counseling), and number of clients with selected outcomes (e.g., GED, unsubsidized employment).

Head Start.     The standard Head Start reports includes information in the broad categories of children enrolled by demographics, staff information by demographics, information on classes/ groups/ centers, volunteer information, and services provided. Head Start also currently has a requirement for extensive outcome measurement. 

Child Welfare Services.     As reported to us, there are no specific standardized reporting requirements; however each grantee must report how they are progressing toward their 5-year plan.

Promoting Safe and Stable Families.     Like other Child Welfare programs, we are aware of no specific reporting requirements beyond reporting concerning progress toward planned activities.

Family Violence Prevention: Grants for Battered Women’s Shelters.     Current Federal reporting requirements include narrative reports on number of clients served and on services provided.

Administration on Aging (AoA) Grants for Native Americans.     The standard report for AoA includes information on full-time/part-time staff; program resources and expenditures, including sources of income other than grants; unduplicated numbers of people that receive support services, congregate meals, home-delivered meals; total numbers of congregate and home-delivered meals; units of supportive services, legal services, at-home services, ombudsmen services, and others.

SAMHSA Targeted Capacity Expansion Grants.     Current Federal reporting requirements include a quarterly report and specified measures related to the Government Performance and Results Act (GPRA).  The quarterly and GPRA reports include the following information: grantee information; staffing information; data including number of new clients, services provided and individual-level information on the clients as required by SAMHSA to meet their GPRA reporting requirements^[12]; and narrative information about the project such as challenges and successes over the past quarter.

3.3 - Methods for Obtaining Input from Technical Working Group and Others

The Evaluation Feasibility Study is just one component of the DHHS efforts to support effective Tribal Self-Governance.  DHHS consulted with Tribes on a variety of issues related to the Title VI Self-Governance demonstration feasibility study that was mandated by Congress to examine potential DHHS programs for Tribal Self-Governance and which has now been submitted to Congress.  A Title VI Advisory Group, comprised of Tribal Leaders with a commitment to Self-Governance and their designees, worked closely with DHHS throughout that earlier process.  In addition, DHHS made information regarding the Title VI Self-Governance demonstration feasibility study available to all interested persons through its website and through direct mailings to Tribes and other stakeholders.

The current Evaluation Feasibility Study has continued this practice of active consultation and information dissemination.  Specific communication activities that were undertaken included:

• A project description, explaining the project and its objectives, was sent to all Chairpersons or Presidents of Federally-recognized Tribes, with a background letter.  The packet sent included contact information for the DHHS Task Order Managers (TOMs) and for each of the Co-Principal Investigators, as well as the DHHS website address where ongoing information about the study and its progress is maintained.  Interested individuals were encouraged to provide comments and inquiries, through telephone or email to the OASPE Task Order Managers and/or the Co-Principal Investigators.
• The Task Order Managers and Co-Principal Investigators conducted presentations and question-and-answer sessions on the project at national or regional AI/AN meetings, including the National DHHS/DOI Tribal Self-Governance meetings in San Diego (November 2002) and Palm Springs (October 2003) and the Alaska Native Health Board meeting in August 2003.
• The Technical Working Group was established and met in February 2003 to review and comment on Draft Reports, the Project Work Plan, and to provide guidance to the project.  Monthly conference calls were conducted with the TWG to discuss interim reports and progress on the project, from March through August 2003.  The Technical Working Group met with the project team on December 9-10, 2003 to review and discuss the Draft Final Report on the project.
• Discussion sessions were arranged and held at the National Indian Health Board conference, National DHHS/DOI Tribal Self-Governance meetings, and NCAI conference to obtain broader Tribal comments and suggestions on the study objectives and preliminary findings.
• At the end of the project, the summary of the Final Report will be mailed to all Tribal Chairpersons with contact information for the OASPE Task Order Managers to encourage further comments and suggestions to DHHS.

The goals of all of these communication and information dissemination activities have been to ensure that the project and its development was conducted as an open and ‘transparent’ process and to glean the maximum useful advice and input from the Tribes in the conduct of the study, while seeking to assure a balanced inquiry and lead to valid and objective advice to DHHS. 

3.4 - Methodology for Site Visits and Tribal Data Review

Site visits were made to five Tribes and one Tribal organization to assess the feasibility of conducting an evaluation of DHHS programs managed under Tribal Self-Governance.  The site visits focused on determining the extent to which there are historical documentation and knowledgeable individuals who are able to provide background and information on the development and goals of Tribal management of Federal programs, the Tribes’ management information systems capabilities, and the availability, sources, and completeness of data on each program managed by Tribes.  The two-day site visits were conducted by a two or three-person team during late June through August 2003. 

While selection of site visit participants depended on availability of willing volunteers, the Technical Working Group and DHHS developed several criteria for selecting Tribes for the study.  These criteria were:

• All Tribes selected should have three or more years of experience with compacting or contracting IHS health programs.
• Preference should be given to Tribes that have experience in managing, under contracts or grants, the DHHS programs determined to be feasible to include in a Self-Governance Demonstration of non-IHS DHHS programs.
• In addition, the sites selected, to the extent possible, were to reflect the following:
• Geographic diversity, reflecting the distribution of the AI/AN population;
• Variation in the size of the Tribe’s population; and
• Variation in economic conditions (i.e., average income, employment levels) of the Tribe.

These criteria were substantially met in the selection of the Tribes that participated in the site visits.

The process developed by OASPE, the Technical Working Group, and the project team for recruitment of Tribes to participate in site visits included:  1) a presentation on the project by the OASPE Project Officer at the Tribal Self-Governance meetings held in late April 2003 in Phoenix; 2) distribution of a letter of invitation to all Tribes to participate, and relevant background materials, at the Tribal Self-Governance meetings; and 3) mailing of the letter of invitation and background materials to all Tribal leaders during the last week in April.  Tribes interested in participating in the site visit were asked to contact the project staff by May 20, 2003 to indicate their interest and/or to obtain additional information.

The site visits were conducted in July and August 2003 to the six Tribes/Tribal organizations listed here:

• Bois Forte Band of Chippewa (Minnesota)
• The Choctaw Nation (Oklahoma)
• Port Gamble S’Klallam Tribe (Washington)
• Yukon-Kuskokwim Health Corporation (Alaska)
• Hopi Nation (Arizona)
• Little Traverse Bay Band of Odawa Indians (Michigan)

3.5 - Methodology for Small Group Discussions

The Small Group Discussions, conducted by Kauffman and Associates, Inc. (KAI), brought together Tribal leaders, experienced Tribal program managers and technical experts in Self-Governance program management to provide feedback and response to preliminary findings and conclusions related to this study.  These discussions provided another means for review and analysis of draft findings and conclusions.  A qualitative analysis of these discussions was conducted to identify “major themes and issues” that emerged across the board.  These major themes informed the study team and the Technical Work Group prior to finalizing reports.

People recruited to participate in the Small Group Discussions included Tribal leaders, Tribal management and technical staff with direct experience in the administration of Self-Governance compacts, including financial managers, management information system (MIS) directors, legal or regulatory analysts, program administrators and related positions. Sign-up sheets were provided prior to each group to make sure we had the appropriate mix of expertise in each session and adequate space to conduct each session.

The Small Group Discussions occurred between September and November 2003.  Locations for these discussion groups included the annual consumer conference of the National Indian Health Board, September 29-October 2, 2003, in St. Paul, MN; the National DHHS/DOI Tribal Self-Governance meetings, October 6-10, 2003 in Palm Springs, CA; and. the annual convention of the National Congress of American Indians, November 16-21, 200, in Albuquerque, NM.  At both the NIHB conference and the DHHS/DOI Tribal Self-Governance meetings, a separate room or break-out session was provided to conduct these discussions.  Rich and substantial qualitative data was collected during these sessions.  Less effective was the session conducted at the NCAI gathering, where this topic was one of several on a busy agenda.  Comment sheets were distributed but few turned in from the NCAI event.  The majority of comments reflected in this report are from the NIHB and the Self-Governance meetings.

Each discussion group involved 10 to 25 individuals representing a mix of interests and experiences from Tribes and Tribal organizations, including both Self-Governance and non-Self-Governance Tribes.  Individuals were recruited through fliers, inserts in conference packets or by appearing on the conference agenda as a workshop option. 

Topic Areas and Prompt Questions:

KAI staff facilitated these discussions.  An overview was provided and a written summary of the Draft Findings and Conclusions distributed for review. The following discussion guide was generally followed, however the flow of conversation generally centered upon three main topics:  (1) reaction to the draft findings and conclusions; (2) omissions in the draft; and (3) best outcomes for this feasibility study.  The following are the questions in the formal Discussion Guide:

1. Facilitator describes what this feasibility study did and did not do:
   • Facilitator explains why this study was done. (purpose)
   • Facilitator describes the difference between an evaluation feasibility study and an actual program evaluation.
   • Q & A regarding this overall study
2. Identification of feasible alternatives for measuring success
   • What are participants’ reactions to the options proposed for measuring success?
   • What other means exist to evaluate success?
   • How would this differ between IHS and other HHS programs?
3. Facilitator reviews the preliminary findings and recommendations
   • Do these preliminary findings and recommendations reflect your experiences with SG?  Give examples of why or why not.
   • What’s missing?
4. What is the best outcome from your perspective for this feasibility study?
   • How do you see these preliminary findings or recommendations impacting future opportunities to expand SG compacting to other programs of HHS?
   • How can this study help local planning for SG?
   • Any other comments?

   4. SITE VISIT FINDINGS

4.1 - Overview

These findings reflect the data collected at the six sites visited.  By agreement with the sites, this report does not present information about a single site by name nor does it compare one site to another or to all others.  Instead, the report provides overarching conclusions based on all six site visits and notes important exceptions to these conclusions. 

Specific findings about the availability of data on the history of Self-Governance and program management as well as management processes is presented below, followed by information on data availability for DHHS programs that may be operated by Tribes under a potential Self-Governance demonstration. Following these discussions is a general summary of the findings as they relate to recommendations for the feasibility of a more quantitative evaluation to document the outcomes and successes of DHHS programs that may be included in a new demonstration.

4.2 - Availability, Accessibility, and Quality of Data on History of Self-Governance and Program Management

When meeting with Tribal members at all six sites these issues were discussed:

• Individuals who have been involved in Self-Governance/Tribal management of Federal programs since these programs were first considered;
• Written documentation and reports that chronicle the initial steps that were taken when the Tribe first considered Self-Governance/management of Federal programs; and
• Individuals who have knowledge and information on the goals/objectives of the Tribe for the program, the extent to which those goals/objectives have been met, and how those goals/objectives have changed over time.^[13]

At all six sites, knowledgeable individuals are available and would be willing to serve as sources of information about the process through which Tribes come to Self-Governance or management of Federal programs. All of the sites indicated that there are individuals in the Tribe who have been involved in Self-Governance/Tribal management of Federal programs since these programs were first considered.  They provided names of these individuals and indicated that they believed that these individuals would be willing to be interviewed if an evaluation were conducted.

All of the Tribes had individuals who were present when Self-Governance and management decisions were being made and would have knowledge and information on the goals/objectives of the Tribe for the program, the extent to which those goals/objectives have been met, and how those goals/objectives have changed over time.

We also found that written documentation prepared for other purposes can serve as a source of information about the steps each Tribe took toward Self-Governance or management of Federal programs.  Specifically, most of the Tribes visited indicated that they had had planning grants or other funding for preparing for Self-Governance or management of programs (especially health) that would provide written documentation of the issues that were considered before the application for management of the program and the key factors that were considered.  Moreover, all Tribes indicated that there were persons available who could describe the structure and operations of Tribal government prior to Tribal Self-Governance/management of Federal programs and the changes that have occurred over time.  In fact, most program-level staff interviewed indicated that they could provide reports and documentation reaching back to the beginning of Tribal management of that program.

4.3 - Availability, Accessibility, and Quality of Data on DHHS Programs That May Be Operated by Tribes Under A Self-Governance Demonstration

In general, information collected during the site visits indicated that all sites currently managing the programs have persons or information available that would help evaluators to  better understand the process that led to Tribal management of these programs.  Moreover, each site indicated to us that they are currently completing all required Federal reporting forms for each program and that these would be available through hard copy or disk from each Tribe for the time period since the Tribe began managing the program. Some Tribes indicated that they were collecting additional information which would also be available.  All Tribes indicated that accounting records were available for these programs beginning with Tribal management. Other program information obtained during the site visits included:

• Tribal Temporary Assistance for Needy Families.  Two of the six Tribes either manage this program or are preparing to manage this program.  Both Tribes indicated that individuals and documentation are available that can provide information on how the Tribe came to manage this program and Tribal goals for this management.  One Tribe uses their overall database of social programs to record client and service information.  This database is by person and records all services received for that person within the center in which the program is housed. This database can also be used to track outcomes.
• Low Income Home Energy Assistance Program.  Three of the six Tribes manage this program.  Levels of record keeping ranged from brief records of services to extensive, very detailed records.  Types of data generally available in varying levels of detail included: number of households assisted, amount of assistance, purpose of assistance. Also, poverty status and age of recipient were available from one site.
• Community Services Block Grant. Two of the six Tribes visited receive Community Services Block Grants.  At both Tribes, records of services included name of recipient, amount, service, and circumstances of needed service.
• Child Care and Development Fund. Five of the Tribes visited manage this program.^[14] Most of the Tribes reported that they used computer software to track the following information: children and families served, hours of childcare, providers, payments to providers, and parent payments.  
• Native Employment Works.  None of the six Tribes manage this program.
• Head Start.  Four of the six Tribes manage this program.  Most of the Tribes visited use computerized by-child records as the basis for their Head Start reports though a few must retrieve all data for reports from hard copy files for each child.  These by-child records are then used to generate summary reports required by Head Start.  Summary reports also require staff and center information that must be retrieved from other records. Hardcopy or disk copies of these reports could be provided on an on-going basis.  All Tribes indicated awareness of the required extensive outcome measurements and reported efforts to comply with that activity. These data would be available for each child.
• Child Welfare Services.  Four of the six Tribes manage these programs.  While a few Tribes maintained computerized databases by child, most indicated that their records are hard copy.  Reports and the data in them for these programs varied widely across the Tribes.  Narrative reports generally listed the number of children served and services provided.
• Promoting Safe and Stable Families.  Only one Tribe managed this program. This Tribe maintains a child registry database for all children served by the center where this program is managed.  The Tribe reported that this database will be used by the Tribe to track outcomes as well as services. 
• Family Violence Prevention: Grants for Battered Women’s Shelters.  Four of the six Tribes receive these grants.  For most Tribes hard copy reports were available.  (One Tribe maintains a child registry database for all children served by the center where this program is managed.)   The content of these hard copy reports varied widely.  
• Administration on Aging Grants for Native Americans.  Five of the six Tribes receive these grants.   Hard copy data on services provided under the grant are maintained by these Tribes.
• SAMHSA Targeted Capacity Expansion Grants.  Only one Tribe visited manages this program and services were provided through the Health Center.

4.4 - Summary of Availability of Data

In general, the site visits conducted indicate that data are available on the relevant DHHS programs and, for these six Tribes, the available data often is much more extensive than the data that are required to be submitted to Federal agencies for reporting.  Key findings include:

• Most Tribes visited have current staff able to provide information about the process and goals of Tribal management or Self-Governance of programs. The depth of knowledge of the history and background of Self-Governance and Tribal management of programs that was provided by Tribal staff who participated in the site visit suggests that there would be no problems encountered in conducting an evaluation that included retrospective information on the background, history and objectives of Tribal Self-Governance and DHHS program management.
• All Tribes visited have detailed accounting information for all programs of interest because accounting services are provided centrally. The Tribes visited provided examples of accounting reports on each program of interest that they managed. These reports were comprehensive and detailed and could provide extensive and useful information for evaluation purposes.
• All Tribes visited would be able to provide evaluators with hard copy or disk copy of the standard reports required by the DHHS for most of the programs they manage.  Many Tribes can also provide much more detailed data than are required by DHHS based on reports they provide to Tribal leaders.

These findings suggest that Tribes have existing financial data systems that would, in most cases, be able to provide considerable financial detail on each of the DHHS programs of interest.  In addition, most Tribes have either hard copy or electronic data on each DHHS program that they manage that could provide baseline information for an evaluation. 

5. DISCUSSION GROUP FINDINGS

The Small Group Discussions were conducted to obtain input and information from a wider group of knowledgeable individuals on the extent to which the draft findings and preliminary conclusions, based on the site visits, “ring true” from the experiences and perspectives of discussion group participants.  Discussions occurred at the NIHB and the Self-Governance conferences. A Summary of these findings and other comments and suggestions is provided in matrix form in Appendix C of this report. Generally, comments fell into the following eleven themes:

• Financial Data:  There was considerable objection to the possibility of examining ‘total Tribal revenues’ as one measure for future evaluations of programs administered through Self-Governance.  While participants could understand why an evaluation might examine expenditures for a specific program, most felt that total Tribal revenues were off-base and outside the scope of any evaluation.  In particular, there were concerns that Tribal gaming revenues would be reviewed along with Tribal program revenues, opening a much larger issue.
• Political Concerns:  There was concern that a political undercurrent among some Tribes could be exacerbated if future evaluations were to make comparisons between Self-Governance and non-Self-Governance Tribal programs.  There was also fear that setting statistical evaluation standards would  increase pressure on Tribes to produce higher numbers, and move Tribes toward regionalization, possibly against the wishes of individual Tribes.  It was also noted that the purpose of Self-Governance is to allow Tribes the authority to develop programs and services based upon Tribal priorities and not Federal or across-program priorities.  This would make across-the-board evaluations of multiple programs difficult.
• Employment:  Some discussants said it was not clear what employment measures would reveal about a program’s success and why ‘turn-over’ is examined instead of employee interviews.  Most discussants also said that Tribes do have salary data by position available in program budgets and were confused about the suggestion in the draft report that these data are not available.
• Data and Measures:  There were concerns that “pre/post” measures must take into account the fact that Tribal programs do not receive the same level of support as States for many programs which may be compacted, such as TANF and Child Support Enforcement.  In particular, States do not usually forward the State match for Federal programs to Tribes. It was also noted that the draft document appears to suggest that an “integrated MIS system” is lacking in Tribal programs, but fails to note that it is also missing in most State and Federal programs as well.  There was concern that Tribes will be held to a higher standard than States for these programs.  The report suggesting pre- and post comparisons to baseline measures should also recommend that the responsibility to collect the data generated by States or Federal systems must remain with the Federal government, and not place Tribes in the position of having to retroactively collect comparison data from States. 
• Systematic Issues:  It should be noted that many Tribes administer programs through Tribal organizations or consortia and data by individual Tribe may be difficult to identify.  It should also be noted that many Tribes taking over Federal programs will lack the infrastructure to be evaluated fairly against State or other programs.  There is also no uniform data reporting system across Tribes, and the report should not assume that such a system exists for making program comparisons across Tribes.^[15]
• Success Stories:  Numerous comments were made about the opportunity to use evaluation process to document success stories, Tribal innovation, and creativity.  A process to identify lessons learned and to share this information with other Tribes getting started would be helpful to overcoming challenges.  A qualitative approach would also allow for Tribal communities to tell their stories and convey their own community and cultural values about each program.  Values such as “local ownership”, “community participation” could be conveyed through this process. 
• Evaluation Purpose/Better Focus:  The purpose of evaluation should be better described in the final report.  It is important that this report communicate that it studied the feasibility of evaluating DHHS programs operated via Self-Governance and not the evaluation of Self-Governance alone.  This is an important distinction. 
• Trust:  It was suggested that the best way to reduce Tribal skepticism about future evaluations would be to convey trust in Tribal programs.  Tribal Self-Governance already begins with an assumption of competence at the Tribal level.  Tribes then can identify Tribal outcomes for future measurement.  There is a fear about evaluation that if a Tribe performs poorly it will lose its Federal funding.  This historic lack of trust should be understood and considered for future evaluation.
• Cost Implications:  Several of the standards suggested in the draft document, such as across-the-board comparisons or integrated MIS systems imply significant costs.  If these standards will be among those used to evaluate DHHS programs administered through Self-Governance, then the Federal government must bear the responsibility to cover these costs.  Tribes should also be provided access to data resources, just as States are provided, for many of these programs, so that Tribal infrastructure can expand with these program requirements.
• Tribal Base:  It is important to understand the base point at which many Tribes are beginning the take over of DHHS programs under Self-Governance.  Many will be starting from a point of inadequate funding.  Many will be using Self-Governance as a way to bring more creativity and collaboration to build up an under-funded Federal program.  Many of the Tribes may already be participating in the data reporting process of the DHHS program and may have data available.
• Limitations:  The limits of conducting evaluation should be well understood.  For example, suggestions in the report about across-the-board designs may not be realistic under Self-Governance once Tribes begin to reprioritize and implement innovative or creative changes.  An across-the-board data pool may no longer exist.  It is also important to understand the challenges of working with small populations and small data pools. 

Based on the input and comments obtained through the discussion groups, the project team made several revisions to the approach to the feasibility study, including:  1) Tribal financial data that could be included in an evaluation design were re-specified as program-specific; 2) evaluation components that examined staffing levels and staff turnover issues were eliminated; 3) an integrated MIS system across all programs was not incorporated into the evaluation design approach; and 4) more emphasis was placed on the costs and burden of data collection on Tribes.  It should be noted however, that the TWG advises that if adequate funding for data systems infrastructures were to be available, Tribes would actively support initiatives to advance development of integrated MIS within their program operations.

The comments and issues raised in the discussion groups were consistent with the guidance that was provided throughout the study by the Technical Working Group members.  Similarly, Tribal staff who participated in the site visits raised many of the same issues.  The agreement among the TWG, participants in the discussion groups, and site visit interviewees provided considerable support and confidence that the project team and DHHS were obtaining consistent information and guidance for the study.

6. EVALUATION FEASIBILITY

6.1 - Overview and Approach

The feasibility of evaluating DHHS programs that may be operated by Tribes under  a Self-Governance demonstration is dependent on a number of factors.  Discussions with the Technical Working Group, Tribal representatives that participated in the discussion groups at national conferences, and representatives from Tribes that participated in the project site visits identified the following issues as important to considerations of feasibility.

6.1.1 - Design of Appropriate Comparison Groups. 

Evaluation methodology requires that the impacts and outcomes of programs being evaluated be compared to the impacts and outcomes that would have occurred in the absence of the new program.  Design of appropriate comparison groups is a critical evaluation feasibility issue. 

Two types of comparison groups are generally used in a rigorous evaluation methodology:  1) pre-post comparisons to examine how the new program differs and what impacts it had, compared to the situation prior to the new program; and 2) external comparisons to control for underlying trends and changes that may affect the program being evaluated and the results produced by the evaluation. 

Many of the Tribes that might participate in a new demonstration program may already be managing the DHHS programs under contracts or grants.  For these Tribes, the pre-post comparison strategy would involve obtaining baseline data on the Tribes’ currently managed programs and then examining changes that occur as the Tribes’ operate the programs under Self-Governance.  If some participating Tribes did not manage specific programs under contracts or grants prior to the demonstration, there may be no “pre-” data for comparison at all or the “pre-” data may be only available for State-managed programs that may be more generously funded or otherwise inappropriate as a baseline for evaluating the program under Tribal management. It would be possible to collect baseline data on the extent to which Tribal members were served by the relevant DHHS programs prior to Tribal management.  The Technical Working Group members stated that this could be an important comparison, because services may not have been accessible or provided prior to the Tribe’s assuming responsibility for program management.

Appropriate external comparison groups may also be difficult to define for similar reasons, but could be constructed based on statistical methodologies developed to account for differences in funding levels and program objectives. Technical Working Group members and discussion group participants emphasized strongly that Tribal programs are often not funded at levels that are comparable to joint Federal-State funding for programs and that this must be accounted for in any design that includes external comparison groups.

6.1.2 - Data Availability. 

Evaluation research requires that data be available for the pre-intervention period, for the post-intervention period, and for appropriate comparison groups.  Based on findings from the site visits and the discussion groups, it is likely that pre-intervention data would be available for  DHHS programs, which might be authorized by Congress for inclusion in a new demonstration, for Tribes that currently manage those programs under contracts.  For Tribes that would choose to participate in the potential demonstration and did not previously manage specific programs under contract, it would be necessary to create a pre-demonstration baseline that could be used to evaluate the DHHS programs managed under a new Self-Governance demonstration. 

In general, it would be possible to develop data collection protocols and strategies to obtain the data necessary for evaluation of DHHS programs managed under a new Self-Governance demonstration.  The complexity and costs of such data collection would vary depending on the specific evaluation issues that were of interest, the unit of observation for which data were desired, and comparison groups that were used to evaluate the programs.

6.1.3 - Costs to DHHS and to the Participating Tribes. 

While it would be possible to design an evaluation of DHHS programs that may be managed by Tribes under a Self-Governance demonstration and to collect necessary data, the costs of the evaluation and data collection activities could be so high as to render the evaluation infeasible.  DHHS has limited funds available for research and evaluation and, if the costs of an evaluation were very large, that would render the evaluation infeasible.  In addition, if a particular evaluation strategy imposed significant costs and data reporting burden on Tribes, it is likely that few Tribes would support the evaluation and volunteer to participate.  Alternatively, if DHHS assumed full responsibility for data collection and reporting costs incurred by Tribes, this would increase the cost of the evaluation to DHHS.

6.1.4 - Trade-offs Between Costs and Usefulness of an Evaluation. 

With any evaluation, the comprehensiveness, number of sites, types of comparisons, and amount of primary data collection affect costs.  A comprehensive evaluation, with a wide range of issues, a large number of sites, both pre-post and external comparisons, and extensive primary data collection would likely be costly, but also produce reliable and defensible results.  A very limited evaluation, with a few priority issues, a limited number of sites, pre-post comparisons, and minimal primary data collection, would be significantly less costly, but might produce fewer useful findings.  

6.1.5 - Tribal Support for a Specific Evaluation Approach. 

The extent to which Tribes may support an evaluation is a key issue for this study. Both the Technical Working Group and most of the Tribal staff who participated in site visits stated that Tribes would, in general, be supportive and positive about an evaluation of DHHS programs that may be operated by Tribes under a new Self-Governance demonstration.  Designing an appropriate evaluation approach that could produce useful findings for the Tribes, as well as for DHHS, is desirable and should take into account the unique issues of Tribes and the principles of Self-Governance.

Many Tribal representatives who contributed to this project emphasized that any evaluation should be structured as an evaluation of DHHS programs managed by Tribes under a Self-Governance demonstration, rather than as an evaluation of Self-Governance.  There is concern that an evaluation of Self-Governance could be construed and/or the findings used to alter the terms or basis of Self-Governance.  To allay those concerns and encourage Tribes to participate in an evaluation, it would be very important to be clear in the stated evaluation objectives that DHHS programs are to be evaluated, rather than Self-Governance.

Discussions with the Technical Working Group and others also stressed that it would be inappropriate to design an evaluation that used a standard set of outcomes to examine DHHS programs operated under Self-Governance.  A principle of Self-Governance is that Tribes should have flexibility to set objectives and design programs to meet each Tribe’s priorities, which may be different than priorities set for Federal programs, generally.  Tribes might be less likely to support an evaluation that set a standard set of outcomes and more likely to support an evaluation that permitted Tribes to set specific and unique program goals that then were examined to determine whether and what extent these goals were achieved.

In addition, it is probable that Tribes might be more willing to participate if:  1) there is a perceived benefit to Tribes from an evaluation, 2) there is extensive consultation on the evaluation objectives, issues, and data that will be collected, and 3) the costs of data collection and reporting are minor or are the responsibility of the Federal government.  Tribes might be more willing to support an evaluation, also, if there were clear and detailed agreements in place that indicate that evaluation data collection/reporting would be limited to the evaluation period and would not continue after that period.  In addition, an evaluation that was structured to report findings across all participating Tribes or large subsets of Tribes would be more likely to encourage participation than an evaluation that would report on individual Tribes.

Technical Working Group members also said that the total costs to DHHS and the Tribes of an evaluation approach, relative to the potential value of the results, would be a consideration for many Tribes.

6.2 - Tribal Consultation in Development of Any Future Evaluation

In the sections below, three illustrative evaluation models are presented and used to assess and discuss the feasibility of an evaluation of DHHS programs that may be managed by Tribes under a new Self-Governance demonstration. It is important to note, again, that this project was intended to provide information helpful to the design of an evaluation. The Evaluation Feasibility study was not designed to produce a definitive evaluation design and methodology.  Any evaluation that might be considered, at some future time, would be developed with a consultation process between DHHS and the Tribes. Results of this Study are intended only to provide information on feasible options for an evaluation and considerations that can be used by DHHS and the Tribes as part of their consultation about the type of evaluation that could be conducted and the range of issues that could be addressed.

6.3 - Description of Three Illustrative Evaluation Models for Assessment of the Feasibility of Evaluating DHHS Programs that May Be Operated by Tribes Under a Self-Governance Demonstration^[16]

Three illustrative evaluation models were developed to provide a structure for assessing the feasibility of conducting an evaluation of DHHS programs that may be operated by Tribes under a Self-Governance demonstration.  These illustrative models range from comprehensive examination of a wide range of issues to a limited examination of targeted priority issues to very limited examination of issues using aggregate reporting data. Each of the three illustrative evaluation models are described in detail in Appendix D to this report, with respect to underlying assumptions, research questions to be examined, comparison group strategies, and data necessary for the evaluation approach. In addition, generic cost estimates were prepared for each of these models. A description and details of these cost estimates are provided in Appendix E.  Below, each of these models is briefly described and feasibility considerations are discussed.

6.4 - Comprehensive Evaluation Model – DHHS Programs That May Be Operated By Tribes Under a New Self-Governance Demonstration

The Illustrative Comprehensive Evaluation Model for DHHS programs that may be operated by Tribes under Self-Governance, if a demonstration were to be authorized by Congress, is designed to be comparable to comprehensive evaluations that have been conducted of other DHHS programs and new delivery strategies.  It would examine the implementation of the demonstration program, operational characteristics and experiences of the program over several years, and would collect data to permit quantitative measurement of processes and outcomes associated with the demonstration.  Both pre-post and external comparison groups would be structured to permit assessment of the impacts of the demonstration, relative to what would have occurred in the absence of the demonstration.  Data necessary for the comprehensive evaluation would be extensive and primary data collection would be necessary to address some of issues of interest. 

In addition to the illustrative model assumptions in Appendix D, an additional assumption was made that is likely to affect the feasibility of this evaluation model:  for each DHHS program managed by a specific Tribe, the Tribe would set two priority goals/objectives.  The evaluation of outcomes would examine whether the Tribe was able to achieve its self-determined goals/objectives, rather than Federally-determined standard sets of objectives being measured across all participating Tribes.  This assumption reflects the uncertainty about whether a set of common goals and objectives would be developed by Tribes that would permit a standard evaluation approach to be implemented.  The Technical Working Group discussed this issues and indicated that it might be possible to identify a limited set of common goals/objectives, but that there was uncertainty on this issue.  The potential complexity of the DHHS-Tribal consultations that would be needed may affect the timeliness of resolving or implementing a demonstration project.

Feasibility considerations with respect to the Illustrative Comprehensive Evaluation Model for a new DHHS demonstration include:

• Availability of Appropriate Comparisons.  Pre-post comparisons would be possible, for those Tribes that managed the relevant DHHS programs under contract prior to the demonstration.  It would be necessary to construct a reliable pre-demonstration baseline for Tribes that did not manage the relevant programs prior to the demonstration.  Primary data collection would likely be necessary to obtain baseline (pre-) information on eligible persons and services needed and obtained prior to the demonstration, for each relevant program. Appropriate and reliable external comparisons would need to be carefully designed to address issues such as greater funding available to State-managed programs, but methodologies could be designed to adjust the comparison group data for differences in funding and other factors.
• Data availability.  The comprehensive evaluation would require extensive data collection during the demonstration period that is considerably beyond the current data collection and reporting submitted by Tribes that operate the relevant DHHS programs under contracts.
• Costs.  The costs associated with a comprehensive evaluation of the potential DHHS programs that may be operated under a new Self-Governance demonstration, including primary data collection to establish baselines, primary data collection through at least three years of the demonstration, and analysis and reporting would likely range from $3 million to $4 million and could possibly be higher.
• Trade-offs Between Costs and Comprehensiveness.  It would be possible to reduce the costs of a comprehensive evaluation by limiting the evaluation to include only Tribes that were managing the DHHS programs under contracts prior to the demonstration.  This would avoid the necessity of primary data collection to establish a baseline for the evaluation for those Tribes that were not previously managing the new programs.  Similarly, the costs could be less if a decision was made to select a subset of Tribes participating in the demonstration –e.g. only evaluate the DHHS programs for a sample of 15 Tribes, rather than the assumed 25.  However, reducing the number of Tribes that are included in the evaluation would reduce the reliability and usefulness of the findings.     These two changes might reduce the cost of the evaluation to between $2 million and $3 million.
• Tribal Support for the Evaluation Approach.  It is likely that some – or most – Tribes would be willing to support and participate in the comprehensive evaluation, if the outcome measures were uniquely set by each Tribe and if costs of participation were low. On the other hand, if the Tribes were required to bear a significant cost for data collection and reporting and/or if Tribes were to be evaluated based on a standard set of Federally-determined outcomes, there would likely be less support for this approach.  The potentially high total costs of a comprehensive evaluation, discussed below, also could reduce Tribal support for this approach.

In summary, it would be technically feasible to conduct a comprehensive evaluation of DHHS programs operated under a Self-Governance demonstration.  However, the associated costs of a comprehensive evaluation would be high – even if it were conducted only for a representative sample of participating demonstration Tribes.

6.5 - Limited Evaluation Model

The Limited Evaluation Model would include qualitative evaluation of implementation and operational experiences of participating Tribes and would attempt to identify effective management strategies and “best practices’ that would be useful to DHHS and to Tribes that are managing or considering managing programs under Self-Governance.  Two outcome measures for each program would be identified by each Tribe, based on its priorities, and data collection and reporting would be limited to the data necessary to assess the extent to which each Tribe achieved its objectives for the selected outcome measures.^[17] Site visits would be conducted to a representative subset of participating Tribes to obtain information on implementation, operations, and innovative programs and “best practices.” During the evaluation period, participating Tribes would also submit narrative annual reports on services provided, persons served, and a limited number of other aggregate program measures.^[18] Comparison strategies would rely on pre-post comparisons.

Feasibility considerations for the Limited Evaluation Model include:

• Availability of Appropriate Comparisons. Pre-post comparisons would be possible, for those Tribes that managed the relevant DHHS programs under contract prior to the demonstration.  It would be necessary to construct a reliable pre-demonstration baseline for Tribes that did not manage the relevant programs prior to the demonstration.  Primary data collection would likely be necessary to obtain baseline (pre-) information on eligible persons and services needed and obtained prior to the demonstration, for each relevant program.
• Data Availability.  Primary data collection would be necessary to establish baseline information for Tribes that did not previously manage the relevant DHHS programs under contracts.  In addition, data collection on two performance indicators/outcomes selected by each Tribe, for each program managed, would be necessary and one round of site visits would be conducted to each of 15-25 Tribes to collect qualitative information on implementation and operational experiences and innovative programs and “best practices.”
• Costs.  The range of estimated costs for the Limited Evaluation Model is from $1.5 million to $2 million, assuming 25 Tribes would be involved in the evaluation and a three-year evaluation timeframe.
• Trade-offs Between Comprehensiveness and Costs.  The potential costs of the Limited Evaluation Model could be reduced if the participating Tribes were limited to those that had managed the relevant DHHS programs under contracts prior to the new demonstration.  Costs could also be reduced if the sample of Tribes to be studied was reduced to 15, rather than 25.  If both of these changes were made, then the range of estimated costs for the Limited Evaluation Model might be reduced to $1 million to $1.5 million.
• Tribal Support for the Evaluation Approach.  It is likely that many or most Tribes would be willing to support and participate in the Limited Evaluation, particularly if data collection was limited to only a few variables for each program and client and if DHHS provided funding, training, software, and technical assistance to the Tribes for data collection.

In summary, the Limited Evaluation Model is technically feasible, would produce useful results, and would involve moderate costs to carry out.

6.6 - Evaluation Model Using Aggregate Reporting Data

The Evaluation Model Using Aggregate Reporting Data is more limited in scope than the Limited Evaluation Model described above. It would rely primarily on data assembled for aggregate periodic reports submitted by Tribes participating in the demonstration. Tribes would submit periodic reports that would be developed through a negotiated process between DHHS and the Tribes, prior to the demonstration implementation. It is also anticipated that additional information for evaluation purposes would be available from DHHS demonstration program officers who would have ongoing contact with each of the demonstration Tribes.^[19] No primary data collection and no client-specific data would be required for this approach. The model was designed to manage costs, provide ongoing reporting of program services, and would require limited effort on the part of the Tribes participating in the evaluation. The comparison strategy would simply be examination of changes in program operations and achievements for each Tribe over the demonstration period.

Key issues for assessing the feasibility of this Evaluation Model based on aggregate reporting data include:

• Availability of Appropriate Comparisons. Comparisons would be conducted of individual Tribes’ operations and reported aggregate outcomes over time, based on the aggregate report data, for each relevant program.
• Data Availability.  Tribes would submit, on a periodic basis, all the aggregate program data necessary for the evaluation. Tribes that are currently operating these DHHS programs under contract arrangements would have systems and experience with the data reporting formats and would continue to submit these reports, for relevant programs, throughout the evaluation period. Tribes that are not currently managing relevant programs under contracts would be provided training and technical assistance in compiling necessary data and completing the reporting formats.
• Costs.  Costs of this Evaluation Model using aggregate reporting data would be relatively low – around $750,000 to $850,000 for a three-year evaluation timeframe. Costs would be primarily for preparing data collection protocols that concatenate the selected items currently being collected, technical assistance to Tribes for data collection, data entry, and analysis and reporting.
• Trade-offs Between Comprehensiveness and Costs.  While the estimated costs of this evaluation approach are relatively low, the evaluation results would provide general information but less detail on innovative management approaches and “best practices.”  Results would primarily be limited to reporting on current program services and clients served, and probably could provide some information on maintenance of effort and different Tribal priorities.
• Tribal Support for the Evaluation Approach:  Based on the site visits and the willingness of site visit participants to provide data and reports for our review similar to that which would be used by this model, it seems likely that many or most Tribes would be willing to support and participate in this evaluation approach.  Willingness to participate would likely be increased, if the aggregate reporting by program that is necessary for the evaluation was clearly indicated as limited to the evaluation study and would be eliminated at the end of the evaluation.

This model is technically feasible and less costly than the other illustrative evaluation models discussed above.  It would provide some useful information, but would not produce results that would be as rigorous or valuable as the comprehensive evaluation or the limited evaluation models.  However, it would be least burdensome to Tribes in terms of data collection and reporting, particularly for participating Tribes that were previously managing the relevant programs under contracts.  This model also might fail to capture changing program objectives that result from tribal re-design since it would be gathering data only on measures that were reported on prior to the Self-Governance demonstration project.

6.7 - Comparison of Feasibility of the Three Illustrative Models

Table 1:  Matrix of Feasibility Issues for Evaluation of DHHS Programs That May Be Operated By Tribes Under a New Self-Governance Demonstration

APPENDIX A: TECHNICAL WORKING GROUP MEMBERS

APPENDIX B: SITE VISIT DATA COLLECTION FORMS

QUALITATIVE DATA COLLECTION FORM

1. History of Tribal Self-Governance/Tribal Management of Federal Programs

   1. Are there individuals in the Tribe who have been involved in Self-Governance/Tribal management of Federal programs since these programs were first considered?  
                                     

      YES

      NO

                                                     
      • IF YES, who are they?
        
      • Are these individuals willing to and available to be interviewed, if an evaluation were to be conducted?                  

        YES

        NO

   2. Are there written documentation and reports that chronicle the initial steps that were taken when the Tribe first considered Self-Governance/management of Federal programs?                                

      YES

      NO

      
      
      • IF YES, would these documents and reports be available to be reviewed, if an evaluation were to be conducted? 

      YES

      NO

      
      
      • For each Tribally-managed program, are there individuals who have knowledge and information on the goals/objectives of the Tribe for the program, the extent to which those goals/objectives have been met, and how those goals/objectives have changed over time?  
        

        TTANF	YES	NO	N/A	IF YES, who are they?
        HEAD START	YES	NO	N/A	IF YES, who are they?
        LOW-INCOME HOME ENERGY ASSISTANCE PROGRAM	YES	NO	N/A	IF YES, who are they?
        COMMUNITY SERVICE BLOCK GRANT	YES	NO	N/A	IF YES, who are they?
        NATIVE EMPLOYMENT WORKS	YES	NO	N/A	IF YES, who are they?
        CHILD CARE AND DEVELOPMENT FUND	YES	NO	N/A	IF YES, who are they?
        CHILD WELFARE PROGRAMS	YES	NO	N/A	IF YES, who are they?
        PROMOTING SAFE AND STABLE FAMILIES	YES	NO	N/A	IF YES, who are they?
        FAMILY VIOLENCE PREVENTION AND SERVICES GRANTS FOR BATTERED WOMEN’S SHELTERS	YES	NO	N/A	IF YES, who are they?
        ADMINISTRATION ON AGING:  GRANTS TO NATIVE AMERICANS	YES	NO	N/A	IF YES, who are they?
        SAMHSA TARGETED CAPACITY EXPANSION GRANTS	YES	NO	N/A	IF YES, who are they?
        HEALTH SERVICES	YES	NO	N/A	IF YES, who are they?

                
      • For each Tribally-managed program, is there written documentation and reports that reflect the issues that were considered before the application for management of the program and the key factors that were considered? 
        
      • For each program, are there periodic written reports and documents that describe the implementation of the program, operational structure and changes over time, and services provided?
        

        TTANF
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        HEAD START
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        LOW-INCOME HOME ENERGY ASSISTANCE PROGRAM
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        COMMUNITY SERVICE BLOCK GRANT
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        NATIVE EMPLOYMENT WORKS
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        CHILD CARE AND DEVELOPMENT FUND
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        CHILD WELFARE PROGRAMS
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        PROMOTING SAFE AND STABLE FAMILIES
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        FAMILY VIOLENCE PREVENTION AND SERVICES GRANTS FOR BATTERED WOMEN’S SHELTERSFAMILY VIOLENCE PREVENTION AND SERVICES GRANTS FOR BATTERED WOMEN’S SHELTERS
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        ADMINISTRATION ON AGING:  GRANTS TO NATIVE AMERICANS
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        SAMHSA TARGETED CAPACITY EXPANSION GRANTS
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A
        HEALTH SERVICES
        	Documentation of issues at application:	YES	NO	N/A
        	Documentation of implementation, operational structure, changes over time, and services provided	YES	NO	N/A

2. Effects of Tribal Self-Governance/Tribal Management of Federal Programs on Tribal Government and Management Processes
   1. Are there individuals in the Tribe who have been involved in Tribal governance and management for a sufficiently long period that they can describe the changes in Tribal governance and management that occurred as a result or in association with the Tribe’s undertaking management of Federal programs?                  

      YES

      NO

      
      IF YES, who are they?
      
   2. Are there written sources of information that describe the structure and operations of Tribal government prior to Tribal Self-Governance/management of Federal programs and the changes that have occurred over time?                                 

      YES

      NO

      
      
   3. What would you suggest as a strategy for assessing the effects of Tribal Self-Governance/management of Federal programs on Tribal government, management, and on community involvement?

QUANTITATIVE DATA COLLECTION FORM

1. What kinds of information are you collecting? 
   

   			COMMENTS	ATTACHMENT
   Do you collect detailed information on your clients/beneficiaries (number served, characteristics)?	YES NO			YES
   Do you collect information about the services provided to each client/beneficiary?	YES NO			YES
   Do you collect information about outcomes for each beneficiary?	YES NO			YES
   Do you have data on number of full-time and part-time personnel and on personnel ‘turnover’?	YES NO			YES
   Do you have cost information by cost component (administrative costs, personnel costs, other costs, by type)?	YES NO			YES
   Do you have records of funding allocations over time?	YES NO			YES
   Do you have records of funding allocations over time?	YES NO			YES

   
   
2. How long have you been collecting this information? 
   
   Are these data available for each year since the Tribe began managing program?

   YES

   NO

   
   
3. How and where is the information stored?
   
4. If electronic storage (as opposed to paper files), how is it entered? 
   
   How frequently is it entered?
   
5. If electronic, what kinds of formatting information and documentation do you have?
   Are file specifications available? 

   YES

   NO

   
   
6. How could these data be provided for an evaluation (e.g., data file of XXX type, generated reports, hard copy for review)?
   
7. What would be involved in getting permission to access these data?
   
8. What kinds of problems exist in these data (e.g., missing data, miscoded data, long time lag between collection and entering)?

APPENDIX C: MATRIX OF ISSUES AND CONCERNS RAISED AT DISCUSSION GROUPS AT NIHB AND DHHS/DOI SELF-GOVERNANCE CONFERENCES

Discussion Guide Matrix
Do the Draft Findings and Conclusions Ring True in Your Experience?

Discussion Matrix
What Omissions Do You See?

Discussion Matrix
What Are Best Outcomes of This Feasibility Study?

APPENDIX D: DESCRIPTION OF ILLUSTRATIVE EVALUATION MODELS

NOTE:   These Models are presented only to illustrate possible approaches to evaluation of DHHS programs that may be managed by Tribes under a new Self-Governance demonstration and to provide a framework for the discussion of evaluation feasibility.  If a future evaluation of DHHS programs operated under a new Self-Governance demonstration would be developed, there would be extensive consultation with Tribes to develop the specific evaluation approach.

1. Comprehensive Evaluation Model:  DHHS Programs that May be Operated by Tribes Under a Self-Governance Demonstration
   1. Objectives
      1. Conduct a comprehensive evaluation of the implementation, process, and outcomes associated with a demonstration of DHHS programs that may be operated by Tribes under Self-Governance. 
   2. Assumptions
      1. Comprehensive Evaluation of Demonstration Programs
         1. 50 demonstration Tribes
         2. Demonstration Tribes may have contracted the programs they are managing under the demonstration prior to the demonstration, or they may elect to manage programs they have not previously contracted.
         3. An Annual Report format would be developed in consultation with Tribes participating in the demonstration and would be submitted by all participating Tribes.
         4. An Evaluation Data Set would be developed in consultation with Tribes. This Evaluation Data Set would include data on characteristics of individual clients/beneficiaries served by each program, services provided/received, and observed process and outcome measures at the individual client/beneficiary level.
         5. A subset of 15-25 Tribes would agree to voluntarily submit the Evaluation Data Set annually, as well as the Annual Report., for each program and for each year of the demonstration
         6. Two rounds of site visits would be conducted to 15-25 Tribes for in-depth evaluation, once during the initial six months of implementation and again approximately 18 months after initiation of the demonstration.
         7. DHHS program offices would provide baseline reports for demonstration programs managed by the Tribes and national benchmark data for all years required.
         8. All participating Tribes would be provided uniform financial reporting formats and Evaluation Data Set reporting formats and would be provided training and technical assistance to ensure comparable and consistent data.
         9. The evaluation would be conducted over a three-year timeframe.
   3. Research Questions to be Examined
      Note:  Specific research questions would be developed in consultation with the Tribes. Based on discussions conducted during the current study, the general research questions that are likely to be identified might include:
      1. Implementation Issues (first 6 months)
         1. What are the characteristics of Tribes that apply to participate in the demonstration?  That are selected to participate in the demonstration? Are these characteristics different from those of Tribes that do not apply?
         2. What factors are reported by demonstration Tribes as influential in their decision to participate?  What was the most important factor in their decision?  What concerns were identified during the decision process?
         3. Was the community involved in the decision to participate in the demonstration?  How was this accomplished?
         4. How was the planning for the demonstration program organized?  Where was responsibility for planning placed organizationally?  Who was involved?
         5. What changes in organization and staffing of each program occurred as a result of the demonstration planning?
         6. What goals/objectives were established for each program during planning?  Were these goals/objectives different from the goals/objectives that had been in place when the programs were contracted?  If so, what are the reasons?
         7. Was there community involvement in setting goals/objectives for each program?
         8. Were changes made in the funding available to each program under the demonstration?  If so, what were the reasons for the changes?
         9. Were changes made that resulted in cost-savings or more efficient use of resources?
         10. What problems were identified during implementation and how were they resolved?
         11. Was the implementation successful?  Why or why not?
      2. Process Questions (six months and throughout the demonstration period)
         1. What changes in programs, staffing, and organization occurred after the initial implementation period?  For each program, what were the reasons for these changes?
         2. Did the demonstration affect overall Tribal management structure and staffing?  Why or why not?
         3. Were there changes in the goals/objectives for each program after the initial implementation period?  If so, what were the reasons for the changes?
         4. How does the Tribe provide oversight and monitoring of each program?  What is the process for addressing problems or issues that are identified through monitoring?
         5. Was there ongoing community involvement in oversight and monitoring of each program?  If so, how was this achieved?
         6. Are goals/objectives for each program met, on a continuing basis?  What factors are important in achieving these goals?  If the goals/objectives are not met, what were the reasons?  What changes were made in response to identifying barriers to meeting goals/objectives?
         7. What are the perceptions of Tribal leaders and program managers of the benefits of Self-Governance, generally, and as a result of the demonstration?  Are there perceived disadvantages of Self-Governance, generally, and for this specific demonstration?
         8. What are the perceptions of Tribal members who receive services from the programs of the benefits and disadvantages of the changes in management and operation of each program?
         9. Were any program changes made to achieve cost-savings and increase efficiency?
         10. Were some program funds re-allocated to other priorities within the Tribe, after the initial implementation period?  How was the decision made?
      3. Quantitative Measures of Process and Outcomes Questions (to be addressed after two years of operation)
         1. Was maintenance of effort achieved?  That is, did each program serve as many people and provide at least the same quantity of services as were available prior to Self-Governance?  If not, what were the reasons?
         2. Did the mix of services provided change under the demonstration, for each program?
         3. For each program, was the Tribe able to achieve at least two quantifiable goals that were established at the initiation of the demonstration program?
         4. For each program, were any changes made in staff levels or types of staff employed? 
         5. Were there changes in the allocation of program funds to personnel, space, materials, administrative costs under the demonstration, compared to the previous contracted program (if the Tribe previously operated the program under contract)?
         6. Did program costs per person receiving services change under the demonstration programs?
         7. Are program users more/less satisfied with services provided under the demonstration program than they were before implementation? 
   4. Comparison Groups for Quantitative Measures Questions
      1. Demonstration
         1. Pre-Post comparisons
         2. Across-site comparisons
         3. National benchmark comparisons
   5. Data Necessary for the Evaluation
      Note:  Specific data needed would depend on the set of evaluation issues and research questions developed in consultation with Tribes.  Likely data needed would include:
      1. 1. DHHS Programs Under Demonstration
         1. Annual Report data for all Tribes participating in the demonstration.
         2. Evaluation Data Set on persons served, age-gender mix, services provided, outcome measures for each year of the demonstration, for each program, for 15-25 participating Tribes
         3. Detailed financial data for demonstration Tribes, baseline through evaluation period, for each program
         4. Detailed data on staffing, for each program
         5. Consumer satisfaction survey of Tribal members receiving services from each program, baseline and second year of demonstration
         6. Two rounds of site visits to 15-25 Tribes to collect qualitative data on implementation process during first six months and again at 18 months to collect information on operational experiences
      2. Other Data Needed
      1. Socio-economic and demographic data for each Tribe (2000 Census)
2. Limited Evaluation Model
   1. Objectives
      1. To design and conduct an evaluation that addresses a limited set of evaluation issues that are identified by the Tribes and DHHS as high priority and valuable to understanding and assessing DHHS programs that may be operated by Tribes under a demonstration.
   2. Assumptions
      1. 50 Tribes participating in new demonstration.
      2. Demonstration Tribes may have contracted the DHHS programs they are managing under the demonstration prior to the demonstration or they may elect to manage programs they have not contracted.
      3. Site visits would be conducted to 15-25 Tribes for in-depth evaluation.
      4. A Minimum Data Set (MDS) would be developed in consultation with Tribes.
      5. All voluntarily participating Tribes would agree to submit this MDS for the baseline (pre-implementation) period and for each year of the demonstration.
      6. Additional data collection would be conducted only for the 15-25 Tribes selected for in-depth evaluation.
      7. DHHS program offices would provide baseline reports for demonstration programs managed by Tribes and national benchmark data for all years required.
      8. All participating Tribes would be provided uniform financial reporting formats and Minimum Data Set reporting formats and would be provided training and technical assistance to ensure comparable and consistent data.
      9. The evaluation would be conducted over a three-year timeframe.
   3. Research Questions
      Specific research questions would be developed in consultation with the Tribes. Based on discussions conducted during the current study, the general research questions that are likely to be identified would include:
      1. What are the overall benefits to Tribes of participating in Self-Governance of Federal programs?
      2. Do the Tribes use the flexibility of Self-Governance to make changes to programs?
      3. How are decisions made about goals of programs and changes that are made to achieve those goals?  To what extent is the community involved in those decisions?
      4. Do the Tribes meet the specific goals that are established for each program?
      5. Are there innovative approaches that are developed by the Tribes that contribute to effective and efficient management of programs and resources?
      6. What problems are encountered?  How are those problems resolved?
   4. Comparison Groups for Quantitative Measures
      1. Pre-post comparisons
      2. National benchmark comparisons
   5. Data Necessary for the Evaluation
      Note:  Specific data needed would depend on the set of evaluation issues and research questions developed through DHHS consultation with Tribes.  Likely data needed would include:
      1. Baseline data on persons served, age-gender mix, services provided, outcome measures, for each program.
      2. Minimum Data Set on persons served, age-gender mix, services provided, outcome measures for each year of the demonstration, for each program.
      3. Detailed financial data on programs operated by demonstration Tribes, baseline through evaluation period, for each program.
      4. Detailed data on staffing, for each program.
      5. Two rounds of site visits to 15-25 Tribes to collect qualitative data on implementation process during first six months and again at 18 months to collect information on operational experiences.
      6. Socio-economic and demographic data for each Tribe (2000 Census)
3. Evaluation Model Using Only AGGREGATE  Reporting Data
   1. Objectives
      1. To conduct a limited evaluation that relies on aggregate periodic reports on programs managed by Tribes under the DHHS demonstration program.
   2. Assumptions
      1. 50 demonstration Tribes.
      2. A set of Annual Report Requirements, including Financial Reporting Requirements, would be developed, in consultation with Tribes.
      3. Participating Tribes would agree to submit these reports for each year of the demonstration.
      4. DHHS demonstration program officers would provide additional qualitative information to the evaluation team on implementation and process for demonstration Tribes, based on their ongoing interactions with the demonstration Tribes.
      5. Agencies responsible for programs included in the demonstration would provide national benchmark data for baseline and for all years of the demonstration.
      6. All participating Tribes would be provided uniform reporting formats and training and technical assistance to ensure comparable and consistent data.
      7. No individual-level analyses would be conducted of program clients/beneficiaries.  All evaluation analyses and program descriptions would be conducted at the aggregate level.
   3. Research Questions to be Examined
      Note:  Specific research questions would be developed in consultation with the Tribes.  Based on discussions conducted during the current study, the general research questions that are likely to be identified might include:
      1. Implementation Issues (first 6 months)
         1. What are the characteristics of Tribes that apply to participate in the demonstration?  Those that are selected to participate in the demonstration? Are these characteristics different from those of Tribes that do not apply?
         2. What changes in organization and staffing of each program occurred as a result of the demonstration planning?
         3. What goals/objectives were established for each program during planning? Were these goals/objectives different from the goals/objectives that had been in place when the programs were contracted?  If so, what are the reasons?
         4. Were changes made in the funding made available to each program under the demonstration?  If so, what were the reasons for the changes?
         5. Were changes made that resulted in cost-savings or more efficient use of resources?
         6. What problems were identified during implementation and how were they resolved?
         7. Was the implementation successful?  Why or why not?
      2. Process Questions (six months and throughout the demonstration period) 
      1. What changes in programs, staffing, and organization occurred after the initial implementation period?  For each program, what were the reasons for these changes?
      2. Were there changes in the goals/objectives for each program after the initial implementation period?  If so, what were the reasons for the changes?
      3. Are goals/objectives for each program met, on a continuing basis
      4. How does the Tribe provide oversight and monitoring of each program?  What is the process for addressing problems or issues that are identified through monitoring?
      5. Were any program changes made to achieve cost-savings and increase efficiency?
      6. Were some program funds re-allocated to other priorities within the Tribe, after the initial implementation period? 
      3. Quantitative Measures of Process and Outcome Issues
      1. Was maintenance of effort achieved?  That is, did each program serve as many people and provide at least the same quantity of services as were available prior to Self-Governance?
      2. Did the mix of services provided change under the demonstration, for each program?
      3. For each program, were any changes made in staff levels or types of staff employed?
      4. Were there changes in the allocation of program funds to personnel, space, materials, contracted services, administrative costs under the demonstration, compared to the previous contracted program and/or national benchmark data?
      5. Did program costs per person receiving services change under the demonstration?
   4. Comparison Groups for Quantitative Measures
      1. Pre-Post comparisons/Patterns over time
      2. National benchmark comparisons
   5. Data Necessary for the Evaluation
      Note:  Specific data needed would depend on the set of evaluation issues and research questions developed by the Tribal Working Group.  Likely data needed would include:
      1. Baseline data on persons served, age-gender mix, services provided, outcome measures, for each program.
      2. Annual Report data on persons served, services provided, outcome measures for each year of the demonstration, for each program.
      3. Annual Report financial data for programs managed by Tribes under the demonstration, baseline through evaluation period, for each program.
      4. Annual Report narrative information on goals/objectives, program changes, problems encountered, and how problems were resolved.
      5. Other Data Needed
         1. Socio-economic and demographic data for each Tribe (2000 Census).

APPENDIX E: COST ESTIMATES FOR EACH ILLUSTRATIVE MODEL

Generic cost estimates were prepared for each of the illustrative models.  This Appendix provides information on the assumptions used for each cost estimate, cost components, and total cost estimates.  These cost estimates are provided to illustrate the potential range of costs and components of costs that might be reasonably expected, for each evaluation model.  Specific cost estimates would depend on the detailed evaluation approach that might be developed by DHHS and Tribes, if an evaluation were to actually be conducted.

A. Comprehensive Evaluation Model

Two sets of cost estimates are provided for the comprehensive evaluation model, one based on the in-depth evaluation involving either 25 or 15 Tribes.  The following assumptions were used to make general cost estimates:

• Three meetings of representatives from Tribes involved in in-depth evaluation per year;
• Data collection by Tribal members at all 50 Tribes participating in demonstration and in-depth data collection by Tribal members at either 25 or 15 Tribes;
• Training and technical assistance for all Tribal data collectors;
• Two site visits per evaluation Tribe during evaluation; and
• Attendance by two evaluation staff at two conferences per year.

The table below presents the cost estimates for the major components of the comprehensive evaluation and a total cost estimate.

Estimated Costs for Comprehensive Evaluation Model

B. Limited Evaluation Model

Two sets of cost estimates also are provided for the limited evaluation model, also based on the in-depth evaluation involving either 25 or 15 Tribes.  The following assumptions were used to make general cost estimates:

• One meeting of representatives from Tribes involved in in-depth evaluation per year;
• Data collection by Tribal members at all 50 Tribes participating in demonstration and in-depth data collection by Tribal members at either 25 or 15 Tribes;
• Training and technical assistance for all Tribal data collectors;
• One site visit per evaluation Tribe during evaluation; and
• Attendance by one evaluation staff member at two conferences per year.

The table below presents the cost estimates for the major components of the limited evaluation and a total cost estimate.

Estimated Costs for Limited Evaluation Model

C. Evaluation Model Using Aggregate Reporting Data

The following assumptions were used to estimate costs for the evaluation model suing aggregate reporting data:

• Three meetings of representatives from Tribes during course of study;
• Data collection by Tribal members at all 50 Tribes participating in demonstration;
• Telephone training and technical assistance for all Tribal data collectors;
• No site visits during evaluation; and
• Attendance by one evaluation staff member at one conference per year.

The table below presented the estimated costs for this model by component and a total estimated cost.

Estimated Costs for Evaluation Model Using Aggregate Reporting Data

APPENDIX F:  FEASIBILITY OF EVALUATING INDIAN HEALTH SERVICE PROGRAMS OPERATED BY TRIBES UNDER SELF-GOVERNANCE

The Scope of Work for the contract to conduct the Tribal Self-Governance Evaluation Feasibility Study included requirements to assess the feasibility of conducting an evaluation of Indian Health Service programs operated by Tribes under Self-Governance compacts, as well as the feasibility of an evaluation of DHHS programs that may be operated by Tribes under a potential Self-Governance demonstration.  Early in the study, the Technical Working Group for the project indicated that an evaluation of IHS programs operated under Self-Governance was fundamentally different from an evaluation of a new demonstration program.  The TWG emphasized that Self-Governance of health programs is not a demonstration program and Tribes have been operating IHS programs under compacts for over a decade.  Any evaluation that would be conducted would be retrospective in nature and subject to more limitations than would evaluation of a new demonstration.

In addition, the extensive experience with Self-Governance of IHS programs means that Tribes have had many years to develop management and operational systems and are confident that these programs are successful and effective.  The Indian Health Service also has monitored and worked with current compacted Tribes over this period and the program is well-established.

Because nearly all Tribes report data to the Resource and Patient Management System (RPMS) maintained by the IHS, there are extensive data available to conduct a quantitative evaluation of processes and outcomes of Self-Governance. The Indian Health Service Resource and Patient Management System database is an IHS-wide system designed to provide detailed and comprehensive clinical information for assessment and management of IHS performance.  It has several components for reporting detailed information on patient characteristics, diagnoses, and specific services provided.  The data included on the Ambulatory Patient Care System component are collected and entered daily, one record per ambulatory medical visit, and includes information on diagnosis, treatments, and specific examinations and tests performed and their results.  In addition, results of special medical record audits are entered into the database.  The RPMS has the capability to produce special reports, by IHS Region, Reservation, Service Unit, and by individual physician.  A pre-post comparison strategy and/or an external comparison strategy based on direct service IHS facilities would be possible, assuming reliable and consistent RPMS data were available for the time frame that Self-Governance has been in place.  However, the Indian Health Service has not conducted such an evaluation.

In this Appendix, we present the background information and evaluation feasibility findings for Indian Health Service programs operated under Self-Governance compacts.  Information on which the evaluation feasibility study was based includes a review of IHS RPMS data capabilities and site visit findings.  Results of these two activities are described below.  Then, illustrative models for the evaluation of IHS program Self-Governance are described and reviewed for feasibility.

Indian Health Service Data Review

Several IHS staff was interviewed about data available through the Resource and Patient Management System (RPMS).  There were two primary foci in these discussions:  patient-level data and administrative and personnel-related data.  The RPMS is an integrated software system for management of clinical and administrative data in IHS and Tribally operated healthcare facilities.  It is composed of several different data collection components.  The Patient Care Component (PCC) comprises data collected at the patient level regarding all care received through the service units and includes a number of client characteristics.  Among the data elements that were investigated and found to be available in some form from the RPMS, and particularly the PCC, from 1998 forward are those listed in the Table F1 below.

Table F.1:  Selected data available through the Patient Care Component of the RPMS

The availability of administrative and personnel-related data elements was also investigated.  These elements would include staffing information, information on staff credentials, staff turnover, pharmacy information, and payment information.  Some of this information are likely available through other components of the RPMS (e.g., accounts payable, contract health, staff credentials), if these components are in use by the Tribal entities of interest. 

Site Visit Findings: Availability, Accessibility, and Quality of Data on Self-Governance of IHS Health Programs

Four of the six Tribes visited have IHS compacts and two of the Tribes have 638 contracts.  The Tribes have managed components of health care for a minimum of four years. Each of the sites indicated that individuals knowledgeable about the Tribe’s experience in Self-Governance or management of health care are available and could provide historical background and other information for an evaluation.  

Accounting and personnel data concerning health programs were available at all sites.  Detailed accounting data were available at all sites including cost information by cost component (e.g., administrative costs, personnel costs, and other) and funding allocations by source over time.  Personnel data including staff turnover information are available at all six sites but are not generally available electronically.  This information would, in most cases, have to be recreated by knowledgeable managers.

All six Tribes use the RPMS system for collecting patient-care data. A few Tribes were also using RPMS data for third party billing or looking into using RPMS for third party billing activities.  Follow-up information gathering with the Indian Health Service revealed that most Tribes/Tribal organizations with Self-Governance compacts (78 of 81 compacts) do submit data to the RPMS.

However, a few of those interviewed expressed concern over the quality of the data collected through the RPMS.  These Tribal interviewees suggested that, without intensive effort at the facility-level to enhance quality, the data were not extremely useful.  One site had enhanced the RPMS data by training staff members in its use, collecting additional outcome data, and conducting a separate patient satisfaction survey.

Illustrative Evaluation Models for Indian Health Service Programs Operated Under Self-Governance Compacts

Illustrative Comprehensive Evaluation Model – IHS Health Programs

A comprehensive evaluation of Indian Health Service programs managed by Tribes under compacts would involve examination of implementation and operational experiences and analysis of the impact of Tribal management on process and outcomes.  As with the illustrative comprehensive evaluation of DHHS programs that may be operated by Tribes under a potential Self-Governance demonstration, Tribes would identify unique health program objectives and quantitative evaluation of processes/outcomes would examine these unique objectives for each Tribe.

The comprehensive evaluation of IHS health programs managed under compacts could be conducted as a separate evaluation, or could be conducted as part of a comprehensive evaluation of all DHHS programs managed by Tribes under Self-Governance.  In the latter case, a subset of Tribes that participate in the new demonstration program and that currently compact for health could be used to evaluate health programs managed under compacts.

Tribes have been managing their health systems under compacts for a decade, so an evaluation of implementation of these programs and operational experiences would necessarily be retrospective in nature.  The retrospective nature of the evaluation might introduce some biases in the findings, but could provide useful information and insights for DHHS and Tribes.  Because nearly all Tribes and Tribal organizations that compact for health services report data to the IHS Resource and Patient Management System and these data are available for the pre-compact and post-compact period, little primary data collection would be necessary.

With respect to the feasibility issues of interest:

• Tribal Support for the Evaluation Approach.  Tribes that manage health programs under compacts may not be supportive of a comprehensive evaluation of these programs, because they have been managing them for a number of years and they view health program Self-Governance as established and successful. Even if the evaluation was limited only to implementation and operational experiences and an evaluation of the extent to which Tribes achieved specific Tribally-set objectives, it is uncertain whether most Tribes would support or consider such an evaluation useful.
• Availability of Appropriate Comparisons.  Assuming that the IHS RPMS data are available for participating Tribes and for the past 10 years, pre-post comparisons would be possible.  External comparisons could be based on a selected sample of IHS direct service Tribes or on aggregate IHS data that includes both compact and direct service Tribes.  However, either of those alternative external comparison groups is likely to raise concerns about appropriateness of the comparisons and value of the potential results.
• Data Availability.  The availability of the RPMS and other IHS data would make it feasible to conduct the comprehensive evaluation of DHHS health programs and little primary data collection would be required.  Some people with whom the project team discussed the study expressed concerns, however, about the quality and completeness of the RPMS data.  It is possible that substantial work would be required to create the evaluation data base to ensure that the quality of the data was sufficient to produce reliable evaluation results.
• Costs.^[20] Assuming that the only primary data collection was to obtain information on satisfaction and experiences of patients using Tribally-managed and direct service facilities and site visits to 25 Tribes, and that the RPMS and other IHS data were available and usable, the evaluation of DHHS health programs managed under compacts could be conducted by an external evaluation contractor at a cost of approximately $750,000 to $1,000,000.
• Trade-offs Between Comprehensiveness and Costs.  The costs of a comprehensive evaluation of DHHS health programs could be less if a decision was made not to collect primary data on patient satisfaction and experiences.  It also could be reduced if a smaller sample was examined for the evaluation (e.g. 15 Tribes rather than 25).  If both of these changes were made, the evaluation costs might be reduced to $500,000 to $750,000.

In summary, an evaluation of DHHS health programs operated by Tribes under compacts is technically feasible and the cost of such an evaluation would be moderate.  Given that Self-Governance of health programs has been in place for over a decade, however, Tribes might consider that the value of the evaluation findings could be limited.

Illustrative Limited Evaluation Model

A limited evaluation of Self-Governance of health programs could be designed that would focus on a few targeted issues that were identified by the Tribes and IHS as potentially useful.  Under the limited model, a few health indicators might be selected by compact Tribes and RPMS data could be used to identify compact Tribes that have been particularly successful in improving outcomes and meeting goals for those indicators.  Then, site visits could be conducted to identified Tribes and in-depth examination of their programs and processes that have been developed to achieve the successful outcomes that they have demonstrated.  These “best practices” and innovative programs could then be detailed and made available to other Tribes that might want to implement them.

It is likely that more Tribes would support a limited evaluation of this type since it would produce results that could be useful to other Tribes and could have the potential to improve health outcomes through dissemination of information on other strategies.  This approach would not require a formal comparison strategy and only RPMS and site visit data would be required to conduct the evaluation.  Costs of this type of limited evaluation would be modest – around $250,000 to $500,000 if an external evaluator conducted the evaluation.

Illustrative Evaluation Model Using Existing Aggregate Reporting Data

Self-Governance Tribes submit periodic narrative and financial reports to the Indian Health Service on their programs.  These reports could be used to construct profiles and trends over time on activities, objectives, and financial status of compact Tribes.  These limited data could provide information for an aggregate evaluation that would describe the experience over time of compacting.  It also might be possible, using these reports and other information within IHS to examine differences between Tribes that have successfully operated health programs under compacts for a number of years and Tribes that may have initiated planning or Self-Governance who subsequently withdrew from Self-Governance of IHS health programs.

Tribal support for this approach might be less than for the limited evaluation, both because the potential value of the results would be less and because there could be concerns about using confidential financial data reported to the Indian Health Service for evaluation purposes. Data likely exists for this evaluation approach, but it is not clear whether it could be used for evaluation purposes nor it is certain that the information contained in reports to IHS would be consistent and comparable across compact Tribes.  If the data were available and contained potentially useful, consistent information, the costs of this evaluation approach might be in the $150,000 to $250,000 range.  

Table F.2:  Matrix of Feasibility Issues for Illustrative Evaluation Models for IHS Programs Operated by Tribes Under Self-Governance

Appendix G: DETAILED DESCRIPTION OF ILLUSTRATIVE COMPREHENSIVE EVALUATION MODEL: INDIAN HEALTH SERVICE PROGRAMS OPERATED BY TRIBES UNDER SELF-GOVERNANCE

1. Objectives
   1. Conduct a comprehensive evaluation of the operations and outcomes of Indian Health Service health programs managed by Tribes under Self-Governance compacts.
2. Assumptions
   1. Tribes to be included in the evaluation of health programs would be limited to those that have submitted data to RPMS.
   2. Participation of both compact and direct service Tribes in the evaluation would be voluntary.
   3. IHS would provide RPMS data for participating compact and direct service Tribes, for all years required.
   4. Site visits would be conducted to 15-25 compact and direct service Tribes that volunteer to participate in the in-depth evaluation.
3. Research Questions to be Examined
   Note:  Specific research questions would be developed in consultation with the Tribes. Based on discussions conducted during the current study, the general research questions that are likely to be identified might include:
   1. Background Issues (retrospectively)—Health Programs
      1. What are the characteristics of Tribes that compact for health programs?  Are these characteristics different from those of Tribes that do not compact?
      2. What factors are reported by Tribes as influential in their decision to manage their health programs?  What was the most important factor in their decision?  What concerns were identified during the decision process?
      3. Was the community involved in the decision to manage health programs?  How was this accomplished?
   2. Process Issues (retrospectively)
   1. What changes in programs, staffing and organization have occurred since the Tribe began managing the health program?  What were the reasons for these changes?
   2. Were there changes in the goals/objectives for the health program/system after the initial implementation year?
   3. What oversight and monitoring of the health system does the Tribe conduct?  What is the process for addressing problems or issues that are identified?
   4. Is the community involved in decisions made about the health system?  How?
   5. Have the goals/objectives for the health system been met on an ongoing basis?
   6. Have program changes been made to achieve cost-savings and/or increase efficiencies?
   7. Have any health program funds been re-allocated to other priorities within the Tribe?  How was the decision made and who were the decision makers?
   8. What are the perceptions of Tribal leaders and others about the benefits of Tribal management of health programs?  Are there any perceived disadvantages?
   3. Quantitative Measures of Process and Outcome Issues – Health
   1. Do health programs managed by Tribes provide the same level and mix of services as would be available if the Tribes did not manage these programs?  If not, what are the reasons?
   2. Did the health program achieve at least two quantifiable goals, established by the Tribe, during the past three years?
   3. Does the Tribe employ directly all health providers that work at the health unit? If so, has this been the case since the Tribe began managing the health program?
   4. Has there been little or much staff turnover since the Tribe began managing the health program?
   5. Have there been changes in the allocation of program funds to personnel, space, materials, contract health services, administrative costs since the Tribe began managing the health program?
   6. Have program costs per person receiving services changed more/less than would be expected based on IHS funding levels?
   7. Are health program users more/less satisfied with services provided by the health program under Tribal management than are users who receive services through IHS direct service arrangements?
4. Comparison Groups for Quantitative Measures
   1. Pre-post comparisons
   2. Self-Governance versus direct service comparisons
   3. National benchmark comparisons
5. Data Necessary for the Evaluation
   Note:  Specific data needed would depend on the set of evaluation issues and research questions that would be developed through DHHS consultation with the Tribes. A set of possible data needs is described here.   
   1. RPMS data for pre-post compact period, for participating Tribes
   2. RPMS data for pre-post compact period, for participating direct service Tribes
   3. Detailed financial data on health programs managed by participating Tribes, baseline through evaluation year.
   4. Detailed data on staffing, staff-mix, salaries, credentials, and turnover of personnel for participating compact and direct service Tribes, since the beginning of health Self-Governance.
   5. Health care satisfaction survey of Tribal members (on/near Reservation), for participating Tribes managing health programs and for a sample of participating direct service Tribes.
   6. One round of site visits to participating Tribes managing health programs and participating direct service Tribes/IHS service units to collect qualitative data on operations, processes, goals/objectives, perceptions of advantages/disadvantages of Tribal management of health programs.
   7. Socioeconomic and demographic data for all health compact and direct service Tribes

[1] In this Report, the term “Self-Governance” is used to indicate the exercise of opportunities provided under P.L. 93-638, Title IV and Title V, as amended, that may be provided, if Congress authorizes a new demonstration program.  All Tribes inherently exercise self-governance as legitimate governments. 

[2] A site visit was conducted to Yukon-Kuskokwim Health  Corporation (YKHC)  in Alaska. YKHC is a Tribal organization authorized to carry out programs on behalf of Federally-recognized Tribes.  Throughout this Report, “Tribes” is used for convenience to reference both Federally-recognized Tribes and Tribal organizations that operate programs on behalf of multiple Tribes. 

[3] Some Tribes would like to be more inclusive than just the 11 programs identified by DHHS. The bill currently before Congress (S.1696) also includes Mental Health and Substance Abuse Block Grants under Title XIX of the Public Health Service Act and Community Health Center grants under Section 330 of the Public Health Service Act.  It is possible that these and/or additional DHHS programs might be included in a demonstration program, if authorized by Congress.

[4] In the case of the SAMHSA Targeted Capacity Expansion program, a broader program is proposed than what is currently available in the SAMHSA grant portfolio.  The new program would entail a combination of mental health and substance abuse services.

[5] Appendix A to this Report provides the list of members of the Technical Working Group.

[6] DHHS also included in the contractor’s Scope of Work a task to examine the feasibility of evaluating the current DHHS Indian Health Service programs managed by Tribes under self-governance.  Because the issues are substantially different for the feasibility of evaluating an on-going program, rather than a new demonstration program, results of that examination are presented in Appendix F of this Report.

[7] Referenced Reports are available on the DHHS/ASPE website, http://aspe.hhs.gov/SelfGovernance/Evaluation/actions.htm.

[8] The information in this section has been drawn from a number of DHHS sources available on the self-governance website (accessed at http://www.aspe.hhs.gov/selfgovernance/).

[9]  P.L. 102-477 allows Federally-recognized Tribes and Alaska Native entities to combine formula-funded Federal grants funds that are employment and training-related into a single plan with a single budget and a single reporting system.

[10] Federal Register, Department of the Interior, Bureau of Indian Affairs, Indian Entities Recognized and Eligible to Receive Services from the United States Bureau of Indian Affairs, Vol. 67, No. 134, July 12, 2002.

[11] All of the DHHS programs discussed in this section require the submission of Standard Form 269 to report financial data.  The financial data reported are aggregated and would not be sufficient for most evaluation purposes.

[12] SAMHSA’s established GPRA requirements include individual-level measures on drug and alcohol use; family and living conditions; education, employment, and income; crime and criminal justice status; mental and physical health problems and treatment; demographics; follow-up status; and discharge status.

[13] Data collection protocols are presented in Appendix B.

[14] Yukon-Kuskokwim Health Corporation is not a CCDF grantee.  However, two CCDF grantees provide services in Bethel, where YKHC is located:  1) Association of Village Council Presidents; and 2) Orutsararmiut Native Council.  The project team was not able to obtain information on these CCDF programs during the site visit or through follow-up telephone contacts.

[15] Note:  There are reporting requirements for Tribes that manage DHHS programs under contracts that could be viewed as uniform reporting systems.  In addition, most Tribes that compact Indian Health Service programs voluntarily submit data to the IHS RPMS system. 

[16] In addition, a similar set of illustrative models were developed and assessed for evaluation of Indian Health Service programs currently operated by Tribes under self-governance compacts.  These are presented in Appendix F to this Report.

[17] If the Tribes participating in the potential demonstration agreed on a common set of goals/objectives, then the evaluation would focus on assessing these common goals/objectives rather than, or in combination with, each Tribe’s uniquely identified goals/objectives.

[18] This additional data reporting would be limited to the evaluation period and would be discontinued at the end of the evaluation.

[19] This demonstration program officer oversight and monitoring is assumed to be similar to the type of monitoring conducted for the “477” consolidated programs.

[20] A detailed analysis of costs was not prepared for the three illustrative evaluation models for IHS programs managed under Self-Governance compacts.  Since the Indian Health Service has most data for the evaluation in-house, it would be possible for IHS to conduct these evaluation approaches as an internal activity.  The cost estimates presented represent approximated costs of an external evaluation, based on costs of comparable evaluations in terms of data collection and analytic complexity.