The Evaluation Feasibility Study was initiated to provide information to the Department of Health and Human Services on the potential to evaluate DHHS programs that may be managed by Tribes under Self-Governance, if a demonstration program were to be authorized by Congress. Most demonstration programs within the Department frequently are developed with an evaluation component that is designed to assess the program’s operations, processes, and outcomes.
The Department of Health and Human Services determined that examining the potential for evaluating DHHS programs that may be managed by Tribes under a potential new Self-Governance demonstration would be useful. The focus of this planning study was to determine whether it would be feasible to conduct an evaluation that included quantitative measurement of process and outcomes, as well as qualitative information on program objectives, implementation, and operations.
Early in the project, it became clear that designing an evaluation of DHHS programs operated under Tribal Self-Governance was a more complex task than is the norm for other DHHS programs. Tribal Self-Governance is intended to allow Tribes to manage their own programs with flexibility and with minimal requirements for reporting to Federal agencies. The most common approach to evaluation requires substantial data collection that is consistent and comparable across organizations participating in a demonstration and a set of common objectives/outcome measures that may be the focus of the evaluation. The goal of flexibility that is a principle of Self-Governance offers challenges in determining a consistent, comparable set of outcome measures and in defining an appropriate methodology for measuring outcomes across participating Tribes. The nature of the government-to-government relationship between the Federal government and individual Tribes also requires consultation and agreement on the type and the extent of any evaluation program. The Technical Working Group (TWG) stressed the importance of these issues at the initial meeting with the project team in February 2003. In addition, the TWG members also stressed that most Self-Governance Tribes are convinced that Self-Governance is a successful and effective means for delivering services to Tribal members.
Generally, many Tribes appear to be supportive of and willing to participate in any potential evaluation of DHHS programs that may be operated by Tribes under a new Self-Governance demonstration. Guidance provided by the TWG for assessing the feasibility of such an evaluation emphasized that there are a number of issues, however, that needed to be addressed in determining the feasibility of any particular evaluation model.
First, the TWG stated it is important that the design of any evaluation be clear in communicating the objectives and benefits of the evaluation to Tribes and provide assurance that evaluation would not have detrimental effects on the future of Self-Governance. The TWG also recommended that design of any evaluation should take into consideration that the underlying goal of Self-Governance is to offer Tribes flexibility to structure programs to better meet local priorities and, consequently, any evaluation that examines a set of outcomes solely Federally-determined would be inappropriate. Instead, there should be an emphasis on flexibility in setting outcomes to be measured based on Tribal priorities and objectives.
The TWG also noted that no evaluation method should require Self-Governance Tribes to provide more extensive reporting than that required of Tribes carrying out programs under current legal authority. Nor should any evaluation method impose extensive and burdensome data reporting on Tribes, rather than the minimal reporting that is one of the principles of Self-Governance. If the burden on Tribes, and associated costs of data collection, are more than nominal, then an evaluation design should include funding to support the costs. In addition, if a specific evaluation design includes extensive data collection, it is essential that there be clear and specific limitations on the time period for the evaluation and a specified date upon which data collection and reporting related to the evaluation would end.
Finally, members of the TWG noted that Self-Governance is an evolving process with new programs. It may take two or three years for a Tribe managing a new program to identify priorities and develop more effective operational and management strategies. Therefore, any potential evaluation should be designed to take into account the evolution of the program and the fact that Tribal goals and priorities may not be determined until the second or third year.
These issues and considerations guided the development of the evaluation feasibility study and the analysis of the feasibility of alternative evaluation approaches.