Tribal Self-Governance Health Care and Social Services Delivery Effectiveness Evaluation Feasibility Study: Draft Literature Review - Revised Report. VI. Implications for the Tribal Self-governance Evaluation Feasibility Study


Qualitative evidence on the effectiveness or impact of Tribal management of health and social services programs is available and, for the most part, suggests that Tribal programs are operating successfully. However, there is very limited quantitative evidence that does not permit firm conclusions concerning the impact or effectiveness of Tribal management, whether under self-governance compacts or under contractual arrangements. Our review of the literature suggests that the information on these issues that may be extracted from the small number of research studies on Tribal self-governance is limited by data and design considerations. Small sample sizes, poor response rates, and the lack of control groups make it difficult to determine the actual effect of these programs or whether these programs may be successfully replicated in other Tribes.

Most of the studies reviewed employed qualitative techniques, such as key informant interviews, which relied on stakeholders' perceptions to base conclusions about program effectiveness. These qualitative studies offer excellent insight into how various Tribes structure their health and social service programs, the characteristics of Tribal residents participating in these programs, and successes encountered in program implementation. Further, these studies - particularly those that focused on the TTANF programs - effectively highlight how the social and economic conditions on the Reservation, such as the high rate of poverty, high unemployment rates, and the lack of an economic base may pose substantial barriers to achieving the intended goals of these programs. These studies do not, however, provide reliable quantitative evidence on the extent to which and how Tribally-managed health and social service programs have operated to better meet the needs of their members. Moreover, because the small number of studies that directly examined issues of access and quality based their conclusions on interviews or surveys of Tribal leaders or program directors, it is unclear whether the Tribal members that these programs are designed to assist have similar perceptions of these programs' impact.

Technical issues in the design of these research studies further limit the ability to draw inferences and to generalize findings to other Tribes or populations. Several studies, for example, are limited to a small number of Tribes in selected States. To the extent that there are cultural differences and that health, social, and economic conditions differ across Tribes, the program structure, services provided, and effectiveness of Tribal management may also differ. A few studies (such as those that compared the percentage of Tribal TANF participants that are engaged in work activities to that for State TANF programs) actually incorporated comparison groups in their design. Because most studies did not incorporate a comparison group in their design, it is not possible to determine how persons participating in Tribal programs fare compared to how they would have fared if control over these programs were still vested with the federal or State government.

One reason for the limitation of existing studies and reliance on qualitative techniques to examine the Tribally-managed health and social service programs is the limited availability of data. For instance, the GAO (2002) evaluation of community contracting for health services in Alaska was unable to assess changes in service availability as contracts were switched from regional health organizations to the community because of their relatively recent implementation, the limited scope of services covered under these contracts, and the unavailability of data. Similarly, the evaluation of the TTANF program conducted by the Administration on Children and Families (2002) specifically indicated that because of data limitations "it is too early to come to any firm conclusions about the success of TTANF programs in meeting their negotiated work participation rates." Among the data problems cited was the fact that several Tribes have established agreements with the State to transmit data to DHHS; in several cases the actual transmission of data had not yet occurred. The GAO (2002) survey of TTANF program directors suggested that basic data necessary for Tribes to operate TANF programs, such as estimates of the number of American Indians in the State who receive TANF benefits, are often of poor quality.

The limitations of existing research on process, structure, and impacts of Tribal management of health and social services are due to three major issues: 1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not generalizable; and 3) the lack of adequate and comparable data across Tribally-managed programs and between Tribally-managed programs and federal and State managed programs. In addition, Tribes may be reluctant to cooperate and participate in evaluation studies, either because they may not perceive a value to them or because they perceive that there are risks that, if the programs are found to be successful, the federal government may interpret these findings as an indication that reductions in federal funding or program support can be justified.

The evidence drawn from the review of the literature suggests that qualitative research has been the primary approach to evaluating Tribal management of health and social services programs to date. Quantitative research, however, has been very limited in past studies, due to lack of reliable and comparable data for Tribally-managed programs (and other factors). A primary focus of the Tribal Self-Governance Evaluation Feasibility Study, therefore, will be to review and identify potential sources of data that would be adequate to permit a quantitative evaluation of relevant issues. In addition, involvement of the Technical Working Group and other Tribal representatives in designing the evaluation feasibility study may increase the likelihood that the Feasibility Study will provide useful information and a process that will ensure that any evaluation that is eventually conducted will be useful to Tribes.