Evidence on the effectiveness or impact of Tribal management of health and social services programs is weak and clearly insufficient to draw conclusions concerning the impact or effectiveness of Tribal management, whether under self-governance compacts or under contractual arrangements. Our review of the literature suggests that the information on these issues that may be extracted from the small number of research studies on Tribal self-governance is limited by data and design considerations. Small sample sizes, poor response rates, and the lack of control groups make it difficult to determine the actual effect of these programs or whether these programs may be successfully replicated in other Tribes.
Most of the studies reviewed employed qualitative techniques, such as key informant interviews, which relied on stakeholders’ perceptions to base conclusions about program effectiveness. These qualitative studies offer excellent insight into how various Tribes structure their health and social service programs, the characteristics of Tribal residents participating in these programs, and successes encountered in program implementation. Further, these studies – particularly those that focused on the TTANF programs – effectively highlight how the social and economic conditions on the Reservation, such as the high rate of poverty, high unemployment rates, and the lack of an economic base -- may pose substantial barriers to achieving the intended goals of these programs. These studies do not, however, provide reliable quantitative evidence on the extent to which and how Tribally managed health and social service programs have operated to better meet the needs of their members. Moreover, because the small number of studies that directly examined issues of access and quality based their conclusions on interviews or surveys of Tribal leaders or program directors, it is unclear whether the Tribal members that these programs are designed to assist have similar perceptions of these programs’ impact.
One reason for the limitation of existing studies and reliance on qualitative techniques to examine the Tribally managed health and social service programs is the limited availability of data. For instance, the GAO (2002) evaluation of community contracting for health services in Alaska was unable to assess changes in service availability as contracts were switched from regional health organizations to the community because of their relatively recent implementation, the limited scope of services covered under these contracts, and the unavailability of data. Similarly, the evaluation of the TTANF program conducted by the Administration on Children and Families (2002) specifically indicated that because of data limitations “it is too early to come to any firm conclusions about the success of TTANF programs in meeting their negotiated work participation rates.” Among the data problems cited was the fact that several Tribes have established agreements with the State to transmit data to DHHS; in several cases the actual transmission of data had not yet occurred. The GAO (2002) survey of TTANF program directors suggested that basic data necessary for Tribes to operate TANF programs, such as estimates of the number of American Indians in the State who receive TANF benefits, are often of poor quality.
The limitations of existing research on process, structure, and impacts of Tribal management of health and social services are due to three major issues: 1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not generalizable; and 3) the lack of adequate and comparable data across Tribally managed programs and between Tribally managed programs and federal and State managed programs.
Of these three issues, the greatest challenge for the conduct of rigorous quantitative evaluations is the lack of adequate and comparable data. This lack of data is due to a number of factors. First, Tribes who elect self-governance of IHS health programs are not required to report specific and comparable data to IHS that would permit evaluation of outcomes, relative to outcomes of IHS direct service provision. Instead, each self-governance Tribe negotiates with IHS to identify specific measures that are relevant and unique to its community. In fact, one of the positive benefits of self-governance of IHS health programs is the low burden of reporting requirements. For other HHS health and social services programs that are managed by Tribes under contracts or grants, reporting requirements may be limited and may be different from reporting requirements for States and federal program offices, and the data submitted may not be accumulated and maintained in a database that is adequate for research purposes.
Second, the American Indian/Alaska Native population represents only about one percent of the U.S. population. Even very large sample national surveys, that provide base data used for many evaluations, seldom obtain sufficient observations of the AI/AN population to permit reliable estimates of socioeconomic, health status, and other characteristics of this population at the sub-State or Reservation level. As a result, there is often no baseline data that could be used to assess the size and characteristics of the potential eligible population to be served by the Tribally managed program or the services that the affected population received prior to the implementation of Tribal management.
Third, American Indians and Alaska Natives may not be accurately identified in many national datasets. As one example, the Indian Health Service conducted a study of the accuracy of AI/AN race coding on State Death Certificates and found, on average, there was an 11 percent miscoding of AI/AN’s as other races and that, in some States, the proportion miscoded was as high as 47 percent (November 1996).
Finally, for many research purposes, the issues of membership in a federally recognized Tribe and geographic location on or near a Reservation are often critical ones. Indian people may be enrolled members of a federally recognized Tribe, members of a State recognized Tribe, or of AI/AN heritage but not an enrolled member of any Tribe. They may live on or near a Reservation or in areas far from a Reservation. Eligibility for benefits under Tribally managed health and social services programs may be restricted to enrolled members of federally recognized Tribes, or to enrolled members of a specific federally recognized Tribe. At this time, however, there are no reliable data that would permit desaggregation of the AI/AN population by Tribal membership and geographic residence. 
The lack of consistent and comparable program data, the relatively small AI/AN population that makes most national survey databases inadequate for study of this population, the problem of inadequate identification of AI/AN race that calls into question data that are available, and the complexity of defining the potential eligible program population are all major obstacles to designing and conducting a reliable quantitative evaluation of Tribal management of health and social services programs.