Tribal Self-Governance Health Care and Social Services Delivery Effectiveness Evaluation Feasibility Study: Draft Literature Review - Revised Report. III.2 Research Limitations


The limitations of existing research on process, structure, and impacts of Tribal management of health and social services are due to three major issues: 1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not generalizable; and 3) there is a lack of adequate and comparable data across Tribally-managed programs and between Tribally-managed programs and federal and State managed programs.

Most new programs require at least three years to fully develop and put in place effective management processes and to refine and focus the program to meet the needs of the specific population served. In addition to avoiding evaluation of programs that are in the 'start up' phase, it is equally important that evaluation focus on multiple years of program experience since any single year may reflect unusual circumstances (e.g., an influenza epidemic could affect costs and distribution of services for a Tribally-managed health program). Thus, evaluation of any Tribally-managed program that has operated for less than five years is unlikely to produce reliable findings that reflect the long-run impacts and effects of Tribal management.

The fact that each Tribe is unique, with different characteristics and culture, suggests that a "one size fits all" evaluation is not appropriate. Individual Tribes may have different priorities and unique challenges that affect the program structure and goals that are put in place. While some general research issues may be appropriate across all Tribes that are managing a specific health or social service program (e.g., what proportion of the total budget is spent on management costs?), the findings must still be interpreted with an understanding of the Tribes' underlying priorities, circumstances, and unique characteristics.

Of these three issues, the greatest challenge for the conduct of rigorous quantitative evaluations is the lack of adequate and comparable data. This lack of data is due to a number of factors. First, Tribes who elect self-governance of IHS health programs are not required to report specific and comparable data to IHS that would permit evaluation of outcomes, relative to outcomes of IHS direct service provision. Instead, each self-governance Tribe negotiates with IHS to identify specific measures that are relevant and unique to its community. In fact, one of the positive benefits of self-governance of IHS health programs is the low burden of reporting requirements. For other HHS health and social services programs that are managed by Tribes under contracts or grants, reporting requirements may be limited and may be different from reporting requirements for States and federal program offices, and the data submitted may not be accumulated and maintained in a database that is adequate for research purposes.

Second, the American Indian/Alaska Native population represents only about one percent of the U.S. population. Even very large sample national surveys, which provide base data used for many evaluations, seldom obtain sufficient observations of the AI/AN population to permit reliable estimates of socioeconomic, health status, and other characteristics of this population at the Tribal or Reservation level. As a result, there is often no baseline data that could be used to assess the size and characteristics of the potential eligible population to be served by the Tribally-managed program or the services that the affected population received prior to the implementation of Tribal management.

Third, American Indians and Alaska Natives may not be accurately identified in many national datasets. As one example, the Indian Health Service conducted a study of the accuracy of AI/AN race coding on State Death Certificates and found, on average, there was an 11 percent miscoding of AI/AN's as other races and that, in some States, the proportion miscoded was as high as 47 percent (November 1996).

Fourth, for many research purposes, the issues of membership in a federally recognized Tribe and geographic location on or near a Reservation are often critical ones. Indian people may be enrolled members of a federally recognized Tribe, members of a State recognized Tribe, or of AI/AN heritage but not an enrolled member of any Tribe. They may live on or near a Reservation or in areas far from a Reservation. Eligibility for benefits under Tribally-managed health and social services programs may be restricted to enrolled members of federally recognized Tribes, or to enrolled members of a specific federally recognized Tribe. At this time, however, there are no reliable data that would permit disaggregation of the AI/AN population by Tribal membership and geographic residence.(3)

Finally, for non-IHS Tribally-managed programs, there may be no relevant data at the Tribe level for the pre-Tribal management period. Prior to Tribes assuming management responsibilities for programs such as Head Start or TANF, these programs were managed by State or local governments and there are seldom detailed data available to assess the level and type of services that were provided to AI/AN people, by Tribe, during the pre-Tribal management period. This lack of 'pre-' Tribal management data and the number and complexity of program management structures and data compilations for the pre-Tribal management period would make it very challenging to design an evaluation with pre-post comparisons for these programs.

One additional factor that may affect the feasibility of evaluation of Tribal management of health and social services program should also be noted. The cooperation and participation of Tribes in an evaluation are critical to feasibility. However, the long history of distrust of the federal government may make that cooperation and participation difficult to obtain. As one participant in the Technical Working Group for the Tribal Self-Governance Evaluation Feasibility Study stressed, Tribes may be concerned about the risks associated with an evaluation - even if the evaluation findings are very positive, they could have a negative impact on future funding of Indian programs.