A comprehensive and rigorous evaluation of processes, structure, and outcomes associated with Tribal management of health and social services programs would use both qualitative and quantitative analyses to address the issues important to understanding the benefits of Tribal management and the factors that contribute to the success of Tribal self-governance/management.
Qualitative methods - key informant interviews, site visits, surveys of perceptions - can provide useful insights and understanding of research questions such as:(2)
- What are the goals of Tribes that manage health and social services programs?
- How do Tribes use self-governance to increase management and internal governance capacities?
- How are programs structured differently under Tribal management?
- What changes are made in the programs, and why were these changes made?
- What problems were encountered in establishing Tribally-managed programs? How were these problems resolved?
- How are community members involved in defining priorities and providing input to guide programs?
- What are the recommendations of Tribal leaders, Tribal program managers, and Tribal program staff that could help them improve services and manage more effectively?
- To what extent has Tribal self-governance of programs resulted in reductions in federal program staff and management costs?
In support of qualitative and process analyses, it is also important to identify and review historical documentation of implementation activities and records of decision-making related to the qualitative outcomes that are identified.
Quantitative methods - are necessary to evaluate the outcomes associated with Tribal management of health and social services programs and to understand the factors that contribute to successful programs. Consistent, reliable, and comparable data are necessary to examine research questions on the impact of Tribal management on measurable performance outcomes, such as:
- How do the numbers and types of services offered change?
- How many people use services, by type? Are services routinely available or is there a significant delay in access or a waiting list?
- What are the outcomes achieved by the program (e.g., percentage of clients receiving preventive health services, increased employment rates)?
- Is the program able to recruit and retain appropriate professional staff to avoid vacancies?
- Is the financial management of the program stable and adequate? Are additional sources of revenues obtained to supplement the base allocation from the federal agency? What are these additional sources of revenue and how much additional funding comes from each?
Evaluation that includes measurement of quantifiable outcomes requires a carefully structured comparison methodology and appropriate data for a sufficiently long time period that outcomes can be accurately attributed as the effects of Tribal management of programs. For Tribally-managed programs that have been operational for a number of years, the most reliable approach to evaluation is:
- Pre-Post Comparisons. A pre-post comparison of quantitative measures (e.g., non-IHS revenues directed to enhancing or increasing services; number of services provided per user) allows the evaluator to assess the change in the measure that can be attributed to Tribal management. Assessment of the impact, however, also requires that the evaluator be able to 'hold constant' other factors. For instance, to assess the impact of Tribal management on number of people served by a specific program requires that the evaluator also take into account other factors that may affect the number of people who want to use these services. If a Tribe's population of children under age 6 is declining, for example, the number of children served by a Head Start program might decline; but the proportion of all eligible children served by a Tribally-managed Head Start program could increase at the same time. Thus, the ability to do a reliable pre-post comparison that takes into account all relevant factors requires: 1) that the Tribe has operated the program for a number of years, so that a stable management system is in place; 2) that data be available for relevant variables for the period before the Tribe took over management of the program; 3) that data comparable to the pre-Tribal management data be available for the years that the Tribe has managed the program; and 4) that the evaluator is able to identify other factors that may affect the quantitative pre-post comparisons and be able to 'adjust' the results to account for these factors.
- External Comparisons. When conducting a pre-post comparison evaluation, it is also very important to take into account underlying trends and changes that affect all Tribes and all programs, whether direct service or Tribally-managed. For example, if medical care cost inflation is 10% nationally, then any evaluation of Tribally-managed health services costs must take into account the underlying inflation rate - rather than assuming that real expenditures rose 10% a year for Tribally-managed health programs. Similarly, if all TANF programs experience an increase in caseloads due to economic recession and higher unemployment rates, then this underlying trend must be incorporated into any evaluation of Tribal TANF before the evaluator can assess the true impact of Tribal TANF on number of people served. To 'control' for underlying national trends and factors, it is necessary to be able to obtain appropriate national (or national program) data that can be used to adjust quantitative measures of Tribal management impacts for 'what would have occurred in the absence of Tribal management'.
A comprehensive evaluation requires very careful and rigorous design and appropriate comparisons to ensure that the analysis is capable of identifying real impacts and effects of the program studied. In addition, if there is a quantitative component of the evaluation, the availability, completeness, and quality of the data used for the evaluation is of critical importance. Finally, knowledgeable interpretation of the quantitative results is also of critical importance for an evaluation to provide accurate and useful information on impacts and effects of a program.