A set of preliminary evaluation issues, research questions, and associated data requirements was prepared for review and discussion with the Technical Working Group. The TWG provided guidance on the general structure and issues for an evaluation of Tribal Self-Governance, as well as specific comments and suggestions for evaluation areas, research questions, and data search methods. General guidance for the project from the TWG included:
- It is very important to examine, broadly, the impacts of Tribal self-governance on Tribe's capacities for management, economic development, and employment. The TWG stressed that self-governance empowers Tribes to develop internal capabilities and economies;
- The management processes that emerge to lead self-governance programs, how they are structured and evolve over time, should be included in an evaluation (that is, "what did the Tribe do and how did they do it?");
- The evaluation should look at the totality of Tribal self-governance and Tribal management of programs under contracts and grants, not just one program;
- Because each Tribe is unique, the evaluation feasibility study should be designed flexibly and should allow for use of measures of performance that vary across Tribes being studied. For example, since Tribes may have different priorities for health programs; the evaluation should be capable of assessing the factors that affect Tribes' priorities and of measuring the extent to which the Tribes are able to address individual priorities;
- The evaluation - and the evaluation feasibility study - should include both qualitative and quantitative components. Quantitative measures alone will not provide the information or accurate reflection of the 'success' of self-governance; and
- Most health status outcomes are not appropriate to include in an evaluation of self-governance of IHS health programs. Self-governance has not been operating over a sufficiently long time period for impacts on health and health status to be observed. There are also many other factors that affect health outcomes (e.g. poverty, education, history of inadequate services) and it would be very difficult to fully account for these factors. Health process outcomes, however, may be feasible to include (e.g. proportion of people over 50 who receive influenza immunizations; proportion of people with diabetes with annual foot examinations).
In addition, the TWG reviewed the preliminary set of evaluation issues and provided input and guidance on appropriate areas for examination. Based on the TWG input and the project team's understanding of OASPE's goals for the Evaluation Feasibility Study, the set of potential evaluation issues for examination in the feasibility study include:
- Qualitative Issues: Overall Effects, Management, Implementation and Operations, Community Involvement
- Self-Governance as a mechanism for strengthening Tribes' capacity for management, economic development, employment;
- Factors that affect the decision for self-governance/Tribal management of programs;
- Processes for establishing community priorities for programs;
- Management structure and stability;
- Implementation processes, problems, solutions;
- Operational processes, problems, solutions;
- Involvement of Tribal leadership and Tribal members in program development;
- Perceptions of Tribal leadership and Tribal members on benefits of Tribal self-governance/management of programs and success in meeting specific program priorities and goals; and
- Effective strategies for augmenting and re-allocating financial resources to support specific programs and across all Tribally managed programs.
- Quantitative Issues: Services Provided, Access, Staffing and Personnel Stability, Financing, Quality, Satisfaction
- For specific programs, extent to which the Tribe was successful in achieving its individual goals for the program (e.g. for self-governance of health programs, if the Tribe's priorities were to increase the proportion of women receiving prenatal care during the initial trimester, did this proportion increase?);
- For specific programs, extent to which the type and quantity of specific types of services changed under Tribal management;
- For specific programs, extent to which access to (i.e. number of people served) and use of services (i.e. number of services provided, overall and per user) changed;
- For specific programs, extent to which the number and types of service providers changed under Tribal management and extent to which retention of service providers changed;
- Across Tribes, extent to which there are common process of care measures that demonstrate improvements in services (e.g. for Tribal self-governance of health programs, do more elderly people receive immunization for influenza?;
- For specific programs and across all programs, extent to which Tribes re-allocate resources, obtain supplemental financial support for programs, and become more efficient in using existing financial resources; and
- For specific programs and across programs, extent to which Tribal members served by the program are satisfied with access and service provided by Tribally managed programs.
For the qualitative issues, the Evaluation Feasibility Study team will review the extent to which written documentation of management decisions, implementation, operations, and periodic reporting are available. In addition, the study team will determine the extent to which there are knowledgeable individuals (e.g. Tribal leaders, program directors) who have been associated with self-governance and Tribally-managed programs for a sufficiently long time period that they can provide the evaluation with the history, perspectives, and details of decision-making, priorities, and experiences over the establishment and operation of Tribally-managed programs.
For the quantitative issues, the Evaluation Feasibility Study team will develop a detailed data review protocol and will review, at each site (and possibly submit the protocol to additional Tribes for written responses), the availability of data to address each evaluation issue.
NOTE: THIS DRAFT REPORT WILL BE EXPANDED TO INCLUDE A DETAILED SET OF RESEARCH QUESTIONS AND THE SPECIFIC DATA REQUIREMENTS ASSOCIATED WITH EACH RESEARCH QUESTION, AFTER THE TECHNICAL WORKING GROUP COMPLETES ITS REVIEW OF THE ABOVE EVALUATION ISSUES AND PROVIDES COMMENTS AND ADDITIONAL GUIDANCE.