The Evaluation Feasibility Study was initiated to provide information to the Department of Health and Human Services on the potential to evaluate DHHS programs managed by Tribes under self-governance, both existing health programs and new DHHS programs that might be included in a demonstration. Most demonstration programs within the Department are developed with an evaluation component that is designed to assess the program’s operations, processes, and outcomes. Although Tribal self-governance of Indian Health Service programs has been in place for a decade, no government-sponsored evaluation of that program has been conducted and the limited information available was primarily qualitative. As a result, little information is available on the strategies and processes used by Tribes who compacted for health programs or on the outcomes associated with self-governance of health programs. The Department determined that examining the potential for evaluating potential new self-governance programs, as well as feasibility of evaluating health programs, would be a useful activity. The focus of this planning study was to determine whether it would be feasible to conduct an evaluation that included quantitative measurement of process and outcomes, since the existing research on delivery of services under self-governance had been primarily qualitative.
Early in the project, it became clear that designing an evaluation of DHHS programs operated under Tribal self-governance was a much more complex and daunting task than was the norm for other DHHS programs. Tribal self-governance is intended to allow Tribes to manage their own programs with flexibility and with minimal requirements for reporting to federal agencies. Evaluation, on the other hand, requires substantial data collection and reporting and a degree of consistency in program structure and models in order to facilitate analysis. The nature of the government-to-government relationship between the Federal government and individual Tribes, however, requires consultation and agreement on the type and the extent of any evaluation program. Beyond the consultation process, individual Tribes cannot be required to participate in an evaluation and the decision to participate is determined by each individual Tribe.
The Technical Working Group (TWG) stressed the importance of these issues at the initial meeting with the project team in February 2004. In addition, the TWG members emphasized that there are a number of other issues – both political and practical – that the feasibility study should take into consideration as the project went forward. These included: 1) concerns that an evaluation of self-governance could be interpreted by Tribes and Tribal organizations as an attempt by the Federal government to discredit or end the self-governance programs that already are in place; 2) concerns that, since the underlying goal of self-governance is to offer Tribes flexibility to structure programs to better meet local priorities, evaluation that examines a set of Federally-determined outcomes is inappropriate; 3) concerns that an evaluation may impose more extensive and burdensome data reporting on Tribes, rather than the minimal reporting that is one of the principles of self-governance; 4) concerns that the evaluation data reporting under the demonstration program could become ‘institutionalized,’ and 5) concerns about the potential political ramifications of any evaluation of self-governance of health programs that involved comparisons of compact Tribes with IHS direct service Tribes.
These concerns and issues guided the development of a framework for the evaluation feasibility study. Specifically, the project team addressed the following considerations:
- Any evaluation of DHHS programs operated under Tribal self-governance should be designed to examine how the Tribally-managed program operates to achieve Tribal goals. The focus of an evaluation is not on whether self-governance should continue to be available to Tribes, but rather on how DHHS programs are operated by Tribes under self-governance to address health and social services needs and each Tribes’ priorities and needs.
- An evaluation should be designed with an understanding of the goals and principles of self-governance. Self-governance offers Tribes the opportunity for flexibility to develop and re-structure programs to meet specific Tribal objectives. Any evaluation design should be similarly flexible in defining outcomes that could be measurable.
- If an evaluation were to be conducted, it would be important to clearly state its goals and communicate with Tribes about the potential benefits to them of participation and the possible disadvantages of participation, including burden of data collection and reporting. There should also be clearly defined limits on the timeframe within which evaluation-related data would be requested and submitted by participating Tribes. In addition, confidentiality of individual Tribes’ data and results should be guaranteed.
- An evaluation of new DHHS programs managed by Tribes under a potential demonstration program that may be authorized by Congress should be separate from and designed differently than an evaluation of self-governance of health programs. Self-governance of health programs is not a demonstration and has been in place for a decade. Therefore, any evaluation that could be structured would be retrospective and subject to more limitations than would an evaluation of a new demonstration program.
These issues and considerations guided the development of the evaluation feasibility study and the analysis of the feasibility of alternative evaluation approaches.