Tribal Self-Governance Health Care and Social Services Delivery Effectiveness Evaluation Feasibility Study: Draft Literature Review - Revised Report

Submitted to:
Andrew Rock, Task Order Manager
Office of Planning and Evaluation
Department of Health and Human Services
200 Independence Ave., SW, Suite 447-D
Washington, DC 20201

Delivery Order 27
Under Contract No. HHS-100-97-0017

Submitted by:
Westat
1650 Research Blvd.
Rockville, MD 20850

Authors: Janet Sutton, Ph.D. and Kathryn Langwell of Project HOPE Center for Health Affairs prepared this Report. Sreelata Kintala and Valerie Meiners provided research assistance in the literature search and development of the bibliography.

"

I. Overview and Objectives

I.1 Overview

The Tribal Self-Governance Evaluation Feasibility Study, being conducted by Westat, and its subcontractors, Project HOPE Center for Health Affairs and Kauffman and Associates, Inc., will provide the Office of the Assistant Secretary for Planning and Evaluation (OASPE) with background information and a detailed review of issues, data availability, and data systems that may affect the extent to which a rigorous and defensible evaluation of Tribal Self-Governance of Indian Health Service and other non-IHS programs can be conducted. While a number of assessments of Tribal self-governance programs have been conducted, these have been primarily qualitative in nature. OASPE is interested in determining the feasibility of conducting an evaluation that examines processes and program changes associated with successful self-governance programs, as well as impacts of Tribal self-governance on outcomes, including access to care, services, quality, costs, financial performance and resources, customer satisfaction, and program stability.

This Draft Literature Review represents one component of the background information that is being assembled to provide a foundation for the development of the evaluation issues, and related data requirements, that will guide that design of the feasibility study. The objectives of this literature review include:

• Discussion of data limitations and other factors that pose barriers to conducting comprehensive evaluations of self-governance and Tribal management of health and social service programs.
• Overview of issues and research questions that might be addressed in a comprehensive evaluation of self-governance and Tribal management of health and social service programs.
• Identification of existing studies and evaluations of Tribal self-governance and/or Tribal management of health and social service programs;
• Review of the methodologies and data sources used in previous studies, in order to assess both analytic rigor and generalizability of their findings;
• Synthesis of the available evidence and findings from existing studies; and
• Assessment of the implications of these findings for the Tribal Self-Governance Evaluation Feasibility Study.

In the next section of this Draft Report, the background and definition of Tribal self-governance and Tribally-managed programs are reviewed and a list of the relevant health and social services programs for the literature search and review is provided. Then, a discussion of data limitations and other factors that make comprehensive evaluation of Tribal self-governance and management of health and social service programs challenging is presented in Section III. Sections IV and V present a summary and assessment of the literature on Tribal self-governance/management of health programs and of social programs, respectively. Section VI discusses the implications of the findings from the literature review for the design and approach to the Tribal Self-Governance Evaluation Feasibility Study. Appendix A to the Draft Report briefly describes the literature search methods that were used to identify relevant published and unpublished studies. Appendix B contains a complete bibliography of sources identified and reviewed for this Report.

II. Background on Tribal SELF-GOVERNANCE/TRIBAL Management of Health and Social Services Programs

II.1 Tribal Self-Governance/Tribal Management of Health and Social Service Programs

Tribes currently manage a number of Department of Health and Human Services programs under several arrangements, including self-governance compacts, self-determination contracts, competitive grants, and demonstration programs. Each of these arrangements may provide differing degrees of autonomy to Tribes and may involve different reporting and compliance requirements.

The authority for Tribal management of federally funded programs was initially provided by Congress under the Indian Self-Determination and Education Assistance Act of 1975 (P.L. 93-638).⁽¹⁾ Title I of that Act authorized Tribes to assume management of Bureau of Indian Affairs/DOI and Indian Health Service programs through contractual agreements. From 1975 to the present, Congress has expanded the opportunities for Tribes to manage their own programs and has increased the degree of Tribal authority and discretion in management. P.L. 100-472 amended P.L. 93-638 in 1988 to add Title III, which authorized the Tribal Self-Governance Demonstration Project that allowed Tribes to assume greater control over BIA programs that they managed, including consolidation and re-designs of programs to better meet individual Tribal priorities and needs. In 1992, as part of P.L. 102-477, Congress extended the Title III self-governance demonstration to provide for Tribal self-governance of Indian Health Service programs. Based on the success of these demonstration projects, Congress made Tribal self-governance a permanent program within BIA in 1994 (Title IV), and made permanent Tribal self-governance of IHS programs in 2000 (Title V).

Within the Department of Health and Human Services, Tribal self-governance has been limited to IHS programs. Tribes may choose to manage these IHS programs under Title 1 contractual arrangements or through self-governance compacts under Title V. Bauman et al. (September 1999) point out that there are reasons that some Tribes might choose to contract rather than seek a compact. To receive a Title I contract, the Tribe does not need any prior experience in program management, while evidence of management experience is required for a Title V compact. In addition, the Secretary of DHHS has only a limited time frame within which to accept or decline a Tribe's Title I proposal, while acceptance of a Title V application may take longer. Self-governance compacts offer more flexibility in using funds and re-designing programs and their terms are more flexible and subject to negotiation. However, substantial autonomy and discretion are also permitted for Title I Tribes and, as a result, the advantages of Title V compacts are not dramatically greater than those for Title I contracts.

Tribes manage other DHHS health and social service programs, under contracts, grants, and demonstration programs. These Tribally-managed programs, however, do not generally offer the flexibility of program design and use of funds that self-governance provides to Tribes, and often require extensive application processes and detailed separate reporting requirements.

In the Self-Governance Amendments of 2000 (P.L. 106-260), Congress re-affirmed its commitment to Tribal self-governance. In the Preamble to the Act, the Congress defined the goal of self-governance as "to permit an orderly transition from Federal domination of programs and services to provide Indian Tribes with meaningful authority, control, funding, and discretion to plan, conduct, redesign, and administer programs, services, functions, and activities (or portions thereof) that meet the needs of individual Tribal communities." Specifically, the Congress directed the Secretary of DHHS to "conduct a study to determine the feasibility of a Tribal self-governance demonstration project for appropriate programs, services, functions, and activity (or portions thereof) of the agency [HHS]."

The Office of the Assistant Secretary for Planning and Evaluation conducted the Tribal Self-Governance Demonstration Feasibility Study for Planning and Evaluation, DHHS in 2001-2002. The Draft Report on the Study, released November 5, 2002, identified 11 DHHS programs as "feasible for inclusion in a Tribal self-governance demonstration project". These 11 programs are:

• Administration on Aging
  • Grants for Native Americans
• Administration for Children and Families
  • Tribal Temporary Assistance for Needy Families
  • Low Income Home Energy Assistance
  • Community Services Block Grant
  • Child Care and Development Fund
  • Native Employment Works
  • Head Start
  • Child Welfare Services
  • Promoting Safe and Stable Families
  • Family Violence Prevention: Grants for Battered Women's Shelters
• Substance Abuse and Mental Health Services Administration
  • Targeted Capacity Expansion

There are Tribes currently managing each of these DHHS programs that are recommended for inclusion in a Tribal Self-Governance Demonstration program, under contractual arrangements or grant awards. The Self-Governance Demonstration program, as detailed in the Draft Report, would permit a simpler, multiple-program application process and simpler and consolidated reporting requirements. Most importantly, the Demonstration program would provide "Tribes with the flexibility to change programs and reallocate funds among programs" (p.19) to better address specific Tribal community priorities.

Initiation of a DHHS Tribal Self-Governance Demonstration requires Congressional action prior to implementation. With the prospect that Congress may authorize such a demonstration, DHHS has identified a need to address the absence of conclusive quantitative evaluation to document the successes and outcomes of Tribal management of health and social services programs. DHHS contracted with Westat to conduct the Tribal Self-Governance Evaluation Feasibility Study to provide background information and to assess the feasibility of conducting a rigorous and defensible evaluation of Tribal management of health and social services programs under self-governance compacts and self-determination contracts. This Draft Literature Review provides background information on the current state of knowledge on the processes and outcomes associated with Tribal management - under self-governance compacts, contracts, and grants - of DHHS health and social service programs. The literature search focuses on identifying existing studies of Tribally-managed IHS programs and on studies that have been conducted of Tribal management of the 11 DHHS programs that have been suggested for inclusion in the DHHS Tribal Self-Governance Demonstration program.

III. Research Limitations

III.1 Overview

A comprehensive and rigorous evaluation of processes, structure, and outcomes associated with Tribal management of health and social services programs would use both qualitative and quantitative analyses to address the issues important to understanding the benefits of Tribal management and the factors that contribute to the success of Tribal self-governance/management.

Qualitative methods - key informant interviews, site visits, surveys of perceptions - can provide useful insights and understanding of research questions such as:(2)

• What are the goals of Tribes that manage health and social services programs?
• How do Tribes use self-governance to increase management and internal governance capacities?
• How are programs structured differently under Tribal management?
• What changes are made in the programs, and why were these changes made?
• What problems were encountered in establishing Tribally-managed programs? How were these problems resolved?
• How are community members involved in defining priorities and providing input to guide programs?
• What are the recommendations of Tribal leaders, Tribal program managers, and Tribal program staff that could help them improve services and manage more effectively?
• To what extent has Tribal self-governance of programs resulted in reductions in federal program staff and management costs?

In support of qualitative and process analyses, it is also important to identify and review historical documentation of implementation activities and records of decision-making related to the qualitative outcomes that are identified.

Quantitative methods - are necessary to evaluate the outcomes associated with Tribal management of health and social services programs and to understand the factors that contribute to successful programs. Consistent, reliable, and comparable data are necessary to examine research questions on the impact of Tribal management on measurable performance outcomes, such as:

• How do the numbers and types of services offered change?
• How many people use services, by type? Are services routinely available or is there a significant delay in access or a waiting list?
• What are the outcomes achieved by the program (e.g., percentage of clients receiving preventive health services, increased employment rates)?
• Is the program able to recruit and retain appropriate professional staff to avoid vacancies?
• Is the financial management of the program stable and adequate? Are additional sources of revenues obtained to supplement the base allocation from the federal agency? What are these additional sources of revenue and how much additional funding comes from each?

Evaluation that includes measurement of quantifiable outcomes requires a carefully structured comparison methodology and appropriate data for a sufficiently long time period that outcomes can be accurately attributed as the effects of Tribal management of programs. For Tribally-managed programs that have been operational for a number of years, the most reliable approach to evaluation is:

• Pre-Post Comparisons. A pre-post comparison of quantitative measures (e.g., non-IHS revenues directed to enhancing or increasing services; number of services provided per user) allows the evaluator to assess the change in the measure that can be attributed to Tribal management. Assessment of the impact, however, also requires that the evaluator be able to 'hold constant' other factors. For instance, to assess the impact of Tribal management on number of people served by a specific program requires that the evaluator also take into account other factors that may affect the number of people who want to use these services. If a Tribe's population of children under age 6 is declining, for example, the number of children served by a Head Start program might decline; but the proportion of all eligible children served by a Tribally-managed Head Start program could increase at the same time. Thus, the ability to do a reliable pre-post comparison that takes into account all relevant factors requires: 1) that the Tribe has operated the program for a number of years, so that a stable management system is in place; 2) that data be available for relevant variables for the period before the Tribe took over management of the program; 3) that data comparable to the pre-Tribal management data be available for the years that the Tribe has managed the program; and 4) that the evaluator is able to identify other factors that may affect the quantitative pre-post comparisons and be able to 'adjust' the results to account for these factors.
• External Comparisons. When conducting a pre-post comparison evaluation, it is also very important to take into account underlying trends and changes that affect all Tribes and all programs, whether direct service or Tribally-managed. For example, if medical care cost inflation is 10% nationally, then any evaluation of Tribally-managed health services costs must take into account the underlying inflation rate - rather than assuming that real expenditures rose 10% a year for Tribally-managed health programs. Similarly, if all TANF programs experience an increase in caseloads due to economic recession and higher unemployment rates, then this underlying trend must be incorporated into any evaluation of Tribal TANF before the evaluator can assess the true impact of Tribal TANF on number of people served. To 'control' for underlying national trends and factors, it is necessary to be able to obtain appropriate national (or national program) data that can be used to adjust quantitative measures of Tribal management impacts for 'what would have occurred in the absence of Tribal management'.

A comprehensive evaluation requires very careful and rigorous design and appropriate comparisons to ensure that the analysis is capable of identifying real impacts and effects of the program studied. In addition, if there is a quantitative component of the evaluation, the availability, completeness, and quality of the data used for the evaluation is of critical importance. Finally, knowledgeable interpretation of the quantitative results is also of critical importance for an evaluation to provide accurate and useful information on impacts and effects of a program.

III.2 Research Limitations

The limitations of existing research on process, structure, and impacts of Tribal management of health and social services are due to three major issues: 1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not generalizable; and 3) there is a lack of adequate and comparable data across Tribally-managed programs and between Tribally-managed programs and federal and State managed programs.

Most new programs require at least three years to fully develop and put in place effective management processes and to refine and focus the program to meet the needs of the specific population served. In addition to avoiding evaluation of programs that are in the 'start up' phase, it is equally important that evaluation focus on multiple years of program experience since any single year may reflect unusual circumstances (e.g., an influenza epidemic could affect costs and distribution of services for a Tribally-managed health program). Thus, evaluation of any Tribally-managed program that has operated for less than five years is unlikely to produce reliable findings that reflect the long-run impacts and effects of Tribal management.

The fact that each Tribe is unique, with different characteristics and culture, suggests that a "one size fits all" evaluation is not appropriate. Individual Tribes may have different priorities and unique challenges that affect the program structure and goals that are put in place. While some general research issues may be appropriate across all Tribes that are managing a specific health or social service program (e.g., what proportion of the total budget is spent on management costs?), the findings must still be interpreted with an understanding of the Tribes' underlying priorities, circumstances, and unique characteristics.

Of these three issues, the greatest challenge for the conduct of rigorous quantitative evaluations is the lack of adequate and comparable data. This lack of data is due to a number of factors. First, Tribes who elect self-governance of IHS health programs are not required to report specific and comparable data to IHS that would permit evaluation of outcomes, relative to outcomes of IHS direct service provision. Instead, each self-governance Tribe negotiates with IHS to identify specific measures that are relevant and unique to its community. In fact, one of the positive benefits of self-governance of IHS health programs is the low burden of reporting requirements. For other HHS health and social services programs that are managed by Tribes under contracts or grants, reporting requirements may be limited and may be different from reporting requirements for States and federal program offices, and the data submitted may not be accumulated and maintained in a database that is adequate for research purposes.

Second, the American Indian/Alaska Native population represents only about one percent of the U.S. population. Even very large sample national surveys, which provide base data used for many evaluations, seldom obtain sufficient observations of the AI/AN population to permit reliable estimates of socioeconomic, health status, and other characteristics of this population at the Tribal or Reservation level. As a result, there is often no baseline data that could be used to assess the size and characteristics of the potential eligible population to be served by the Tribally-managed program or the services that the affected population received prior to the implementation of Tribal management.

Third, American Indians and Alaska Natives may not be accurately identified in many national datasets. As one example, the Indian Health Service conducted a study of the accuracy of AI/AN race coding on State Death Certificates and found, on average, there was an 11 percent miscoding of AI/AN's as other races and that, in some States, the proportion miscoded was as high as 47 percent (November 1996).

Fourth, for many research purposes, the issues of membership in a federally recognized Tribe and geographic location on or near a Reservation are often critical ones. Indian people may be enrolled members of a federally recognized Tribe, members of a State recognized Tribe, or of AI/AN heritage but not an enrolled member of any Tribe. They may live on or near a Reservation or in areas far from a Reservation. Eligibility for benefits under Tribally-managed health and social services programs may be restricted to enrolled members of federally recognized Tribes, or to enrolled members of a specific federally recognized Tribe. At this time, however, there are no reliable data that would permit disaggregation of the AI/AN population by Tribal membership and geographic residence.(3)

Finally, for non-IHS Tribally-managed programs, there may be no relevant data at the Tribe level for the pre-Tribal management period. Prior to Tribes assuming management responsibilities for programs such as Head Start or TANF, these programs were managed by State or local governments and there are seldom detailed data available to assess the level and type of services that were provided to AI/AN people, by Tribe, during the pre-Tribal management period. This lack of 'pre-' Tribal management data and the number and complexity of program management structures and data compilations for the pre-Tribal management period would make it very challenging to design an evaluation with pre-post comparisons for these programs.

One additional factor that may affect the feasibility of evaluation of Tribal management of health and social services program should also be noted. The cooperation and participation of Tribes in an evaluation are critical to feasibility. However, the long history of distrust of the federal government may make that cooperation and participation difficult to obtain. As one participant in the Technical Working Group for the Tribal Self-Governance Evaluation Feasibility Study stressed, Tribes may be concerned about the risks associated with an evaluation - even if the evaluation findings are very positive, they could have a negative impact on future funding of Indian programs.

III.3 Implications for Review of the Literature

The lack of consistent and comparable program data, the relatively small AI/AN population that makes most national survey databases inadequate for study of this population, the problem of inadequate identification of AI/AN race that calls into question data that are available, and the complexity of defining the potential eligible program population are all major obstacles to designing and conducting a reliable quantitative evaluation of Tribal management of health and social services programs. In the following two sections of this report, research and evaluation studies of Tribally-managed programs are reviewed and assessed to ascertain the extent to which some studies have been able to address these challenges and provide findings that are informative and useful in understanding the effects of Tribal management of health and social service programs.

IV. Evidence on Tribal Self-governance and Tribal Management of Ihs Health Programs

IV.1 Overview

Evaluation of Tribal self-governance and Tribal management of IHS health program should be considered in the context of the funding limitations facing all IHS funded health programs. Between 1989 and 1998, real per capita IHS spending declined by nearly 20 percent.(4) Similarly, the Indian Health Service estimates that between 1991 and 2001, real per capita expenditures for personal health care decreased by 14 percent - from $1,349 in 1991 to $1,157 in 2001. (5) The "Level of Need Funded (LNF) Study: Research to Estimate Costs of a Mainstream Health Benefits Package for Indian People" (November 1999) reports that, in 1999, personal health care spending for AI/ANs who received services from the Indian Health Service was only 55 percent of the level of need.

The significant under-funding of the Indian Health Service makes evaluation of Tribal management of health programs more difficult than most evaluations. When Tribes take over management of health programs, they do so with inadequate resources from IHS to meet the needs of Tribal members and must immediately assess priorities and develop strategies either to ration care or to obtain additional sources of revenues to provide services. Because Tribes are dependent, for the majority of their revenues, on the IHS, their performance under self-governance may be affected by real declines in the IHS budget. This factor should be taken into account in any evaluation of Tribal management of health programs.

IV.2 Studies of Tribal Self-Governance/Tribal Management of Health Programs

There has been limited quantitative research on Tribal contracting and compacting for IHS health programs or health service delivery. The Indian Health Service Baseline Measures Workgroup, composed of representatives from Tribes and from IHS, addressed the issue of evaluation of Self-Governance of IHS programs and recommended that a two-part evaluation be conducted (Indian Health Service, September 1996). One component of the evaluation would focus on evaluation of the process and outcomes of Self-Governance; the second component would focus on the impact of Self-Governance on non-Self-Governance Tribes. However, no formal evaluation based on these recommendations was subsequently conducted.

Three studies that have examined the impact of Tribal contracting for health services on administration and operations of health programs were identified in our literature search. Each of these studies relied on key informant interviews, site visits, or surveys of Tribal authorities or IHS staff to gather information on the success of Tribal management. Evidence on the effectiveness of self-governance of health programs available from this small number of studies is consistent and positive, but is based on qualitative measures, rather than quantitative measures.

IV.2.1 Access to Care

In terms of program management, the information gathered from the literature indicates that one of the most pressing problems encountered by Tribes operating health programs is difficulty in recruitment and retention of professional staff. In a 1997 survey of approximately 210 Tribes conducted by the National Indian Health Board (NIHB, 1998), approximately 75 percent of responding Tribes that operated their own health programs indicated that they experienced difficulty in recruiting physicians and 40 percent reported difficulty in recruiting mid-level practitioners. In contrast, of respondents affiliated with IHS direct service programs, 67 percent reported that recruitment of physicians and 25 percent reported that recruitment of mid-level practitioners was a problem. Interestingly, Tribes operating their own health programs were less likely than IHS direct service Tribes to report difficulty in recruiting other types of health professionals, such as nurses, pharmacists and dentists.

Little is known about the impact of Tribal contracting or compacting on access to health facilities, providers, and services. There are limited data to suggest that access to some programs may be reduced, while access to other services or programs may actually increase. For example, in the 1997 NHIB survey, 16 percent of compacting Tribes and 32 percent of contracting Tribes indicated that they had eliminated programs during the previous 3-year period. In contrast, 38 percent of IHS health directors responding to this survey indicated that they had eliminated programs during the same time period.

Over the same period, a significant number of contracting/compacting Tribes reported having added services, including:

• One-quarter reported adding mental health services, compared to 14 percent of IHS health directors;
• One-fifth reported that they added alcohol treatment services, compared to 5 percent of IHS health directors who reported adding substance abuse services; and
• Nearly one-quarter reported adding dental services, compared to.10 percent of IHS health directors.

Interestingly, Tribally-managed programs were less likely than IHS programs to have added services for diabetes care. Only 12 percent of health directors of Tribally-managed programs indicated that diabetes care services had been added, compared to nearly 30 percent of respondents from IHS direct service programs.

Although the literature indicates that Tribes either perceive that contract funding is inadequate or are financially struggling to administer health programs (National Indian Health Board, 1998; Noren et al., 1998; GAO 1998) there is little evidence to indicate how access to services may be affected by financial concerns. In the NIHB survey, 20 percent of respondents from Tribally-managed programs indicated that they had to close health facilities; however, less that one-half of these respondents indicated that the closure was related to funding problems. In fact, only 4 percent of Tribes ceased management of health programs altogether. While insufficient budgets were cited as a reason for terminating their participation in contracting/compacting, geographic barriers and regulatory factors were also mentioned as reasons for this decision.

The 1998 GAO study of Tribally-managed health services in Alaska also concluded that service availability was generally unaffected when a community takes over health services from a regional health organization (RHO). RHOs are non-profit entities that contract with the IHS to manage and deliver health services to a Tribe. Instead of having the RHO manage their health service delivery, several Alaska Native communities have chosen to directly contract with the IHS. In transitioning to community control, some service disruptions were noted; however, because contracts between the IHS and communities were generally limited to a narrow set of services (typically alcohol abuse and mental health services, health education and non-physician services), the impact on the community was thought to be minimal. Moreover, since contracts are generally program transfers, where the community takes over from the IHS or RHO the management responsibility for existing services, staffing and services are often unchanged. The GAO cautioned that the availability of contract managed health services, which tend to have higher administrative and indirect costs, may be compromised in the future if funding for contract support services is reduced. (Contract support funds refer to the reasonable costs reimbursed by IHS to Tribal communities to cover contract compliance and program management activities.)

IV.2.2 Quality of Care

There is little quantitative evidence on the quality of care rendered by Tribally-managed health programs and how patterns of care or outcomes may differ from health programs operated by the IHS. Although the NIHB study that was previously discussed did attempt to gather information on quality of care, quality was subjectively measured. In terms of one standard measure of quality - average wait time - 86 percent of Tribal leader respondents from compacting Tribes indicated that they had noted improvements in wait times over the 3-4 years referenced. In contrast, only 41 percent of Tribal leader respondents from contracting Tribes and 19 percent from the IHS direct service programs indicated that wait time had improved over the past years.

Tribal leaders and health directors were asked about their perceptions of the quality of care that the health systems provides to their Tribe and changes in the quality of care between 1993 and 1996. Representatives of contracting and compacting Tribes were more likely than respondents from the IHS to indicate that the quality of care had improved during this time period. Approximately 94 percent of Tribal leaders and Tribal health director respondents from contracting and compacting Tribes perceived an improvement in quality of care compared to only 62 percent for IHS direct service programs.

These findings, of course, are based on subjective perceptions. To date, no independent quantitative assessment of services provided and of quality of care and outcomes has been conducted.

V. Evidence on Tribal Management of Social Services Programs

V.1 Overview

Most Tribal management of non-IHS DHHS social services programs is conducted under contracts that offer less flexibility and independent determination of funding allocations than do self-governance compacts. The exception is multiple programs that can be operated under 477 contracts. P.L. 102-477 gives Tribal governments the option to integrate the employment, training, and related services (e.g. child care services) they operate, even if those services are funded by different federal agencies. 477 contracts allow Tribes to operate multiple programs under a consolidated plan, combining their funds into a single budget, and requiring only one reporting system for all programs. This permits Tribes to have reduced paperwork and also permits re-allocation of program funds to better meet Tribal needs and priorities.(6) In 2001, 43 Tribes and consortia held 477 contracts; of these 12 include Tribal TANF funds and 28 include Native Employment Works funds with those funds from other programs under 477 contracts.

In this section, existing research studies that have examined the impacts and effects of Tribal management of DHHS social services programs are reviewed. No formal evaluation study or research on the impacts of management of these programs within 477 contracts was identified in the literature search.

V.2 Studies of Tribal Management of DHHS Social Services Programs

Even though evidence on the impact of self-governance of IHS health programs is limited, it may be possible to gather insight on the effects of Tribal management of services from the literature on Tribal management of social services and non-IHS health programs. One program for which evidence concerning the impact of Tribal management is available is Temporary Assistance to Needy Families (TANF).

The 1996 Personal Responsibility and Work Opportunity Reform Act (PRWORA), which replaced the Aid to Families with Dependent Children (AFDC) with TANF, included provisions that permitted Tribes to operate their own TANF programs. Under the Tribal TANF (TTANF) program, and unlike State TANF programs, Tribes have the flexibility to establish their own work participation goals and to identify the work-related activities that may meet their self-designated work participation goal. Moreover, whereas State TANF participants are eligible to receive cash benefits for a period of up to 60 months, TTANF programs may determine their own time limits. According to the Department of Health and Human Services, Administration for Children and Families (2002), in 2001 a total of 34 Tribal TANF programs, representing 172 Tribes, had been approved.

Despite a national trend toward decreased caseloads, Tribal caseloads have increased or remained the same since the inception of the TTANF program (GAO, 2002). Research to understand the reasons for this growth and the impact of TTANF programs has focused largely on describing the demographic and socio-economic characteristics of persons served by TTANF programs and the economic conditions (e.g., high unemployment rates, lack of skilled labor), social conditions (e.g., lack of child care or employment supports), and the physical infrastructure (e.g., poor roads, limited public transportation, lack of telephones) that may pose barriers to Tribal implementation of these programs or that prevent TTANF programs from achieving their employment goals (Pandey et al., 1999; Pandey et al., 2000; Pandey et al., 2001; GAO 2002.)

Few studies, however, have directly examined the extent to which Tribal TANF programs are achieving their intended objective of promoting independence through employment. The data on work participation rates that has been reported by the Administration for Children and Families (2002) do indicate that TTANF work participation rates average about 37 percent, with approximately one-third of these individuals engaged in unsubsidized employment, one-third engaged in job search activities, and 8 percent engaged in unpaid work. Other program participants were engaged in TANF eligible activities such as subsidized employment, education and vocational training. Because data on the work participation rates of American Indians who participate in State TANF programs were not reported, it is not possible from this study to ascertain how TTANF programs perform relative to State programs. A GAO (2002) study also noted that, in fiscal year 2001, 43 percent of State TANF recipients were involved in work activities (compared to 37 percent of TTANF recipients). Moreover, 60 percent of persons in State TANF programs were in unsubsidized jobs, compared to only one-third of those in Tribal TANF programs.

Insight on the performance of TTANF programs may be gleaned from a series of studies conducted by staff at the Kathryn M. Buder Center for American Indian Studies (Pandey et al. 1999, 2000, 2001). Staff associated with this Center conducted multiple waves of interviews with members of American Indian families, located on three Arizona Reservations, that were currently or had previously received welfare. A total of 350 persons were included in their second round of interviews. Of the Tribal TANF recipients, 15 percent found employment and exited TANF. Although this figure is substantially lower than national estimates of employment of TANF recipients (23 percent), it represents an increase from the previous survey round, when only 11 percent were able to find jobs and exit TANF.(7) Between the first and second wave of interviews, the proportion of current and former welfare recipients that received employment income increased from 12 percent to 27 percent and the proportion with a checking or savings account increased from 17 percent to 26 percent. Nonetheless, approximately one-quarter of TTANF participants who had transitioned to work were unemployed within three months and one-half were unemployed a year later (Kathryn M. Buder Center for American Indian Studies and Native Nations Institute for Leadership, Management and Policy, 2001).

Closely associated with the Tribal TANF program is the Native Employment Works (NEW) program, which was initiated in 1997 to make work activities or employment opportunities available to Tribal members, and the Welfare-to-Work (WTW) program, which provides funds for welfare recipients to transition to work. Approximately 78 Tribes established NEW programs during the first year of implementation. According to the Administration on Children and Families (2001), despite barriers in implementing NEW programs that included inadequate staffing, limited employment opportunities on or near Tribal lands, inadequate equipment and funding, between 1999 and 2000, 44 percent of persons participating in NEW completed the program after meeting one or more objectives.

Empirical evidence on the impact of the Tribal WTW grant program is unavailable. One of the few studies that evaluated the Tribal WTW (Hillabrandt, 2001) consisted of site visits to 10 of 92 Tribal WTW grantees; Tribes were selected based on the whether they had a "comprehensive and innovative" program. Although this study did not directly evaluate the effectiveness of Tribal WTW programs or the characteristics of programs that may enhance or limit their effectiveness, it did identify barriers to Tribes' implementation of these programs. Among the barriers that Tribes encountered were difficulties in identifying and recruiting participants, difficulties in certifying eligibility, and difficulties in getting people enrolled and encouraging them to continue to participate in the program.

The Tribal Child Care Grant (CCG) program is another social service area that Tribes may manage under contractual arrangements. Again, there is little information on factors associated with effective management or impacts. The CCG program is premised on the assumption that with adequate childcare many unemployed low-income persons could enter the work force. The CCG provides low-income families with child care subsidies. In 2002, over 260 Tribal organizations received a grant under the CCG program (Crow, 2002). As stated, we know little about the effectiveness of the Tribal CCG program. In one of the few studies that have examined this program, the U.S. Department of Health and Human Services, Office of Inspector General (OIG), conducted site visits to 29 Tribes. From interviews with Tribal child care administrators, employees and child care representatives, the OIG concluded that Tribes perceived Tribal management of the CCG program increased access to culturally sensitive child care programs and allowed Tribes to meet the unique needs of their community. However, lack of communication and coordination across State and Tribal programs was thought to lead to increased costs, wasted resources and, in some cases, duplicate payments.

Literature was not found on the impact or effectiveness of other Tribally-managed social services programs administered under the Administration for Children and Families, including Child Welfare Services, Community Services Block Grant, the Promoting Safe and Stable Families Program and Grants for Battered Women's Shelter Program.

VI. Implications for the Tribal Self-governance Evaluation Feasibility Study

Qualitative evidence on the effectiveness or impact of Tribal management of health and social services programs is available and, for the most part, suggests that Tribal programs are operating successfully. However, there is very limited quantitative evidence that does not permit firm conclusions concerning the impact or effectiveness of Tribal management, whether under self-governance compacts or under contractual arrangements. Our review of the literature suggests that the information on these issues that may be extracted from the small number of research studies on Tribal self-governance is limited by data and design considerations. Small sample sizes, poor response rates, and the lack of control groups make it difficult to determine the actual effect of these programs or whether these programs may be successfully replicated in other Tribes.

Most of the studies reviewed employed qualitative techniques, such as key informant interviews, which relied on stakeholders' perceptions to base conclusions about program effectiveness. These qualitative studies offer excellent insight into how various Tribes structure their health and social service programs, the characteristics of Tribal residents participating in these programs, and successes encountered in program implementation. Further, these studies - particularly those that focused on the TTANF programs - effectively highlight how the social and economic conditions on the Reservation, such as the high rate of poverty, high unemployment rates, and the lack of an economic base may pose substantial barriers to achieving the intended goals of these programs. These studies do not, however, provide reliable quantitative evidence on the extent to which and how Tribally-managed health and social service programs have operated to better meet the needs of their members. Moreover, because the small number of studies that directly examined issues of access and quality based their conclusions on interviews or surveys of Tribal leaders or program directors, it is unclear whether the Tribal members that these programs are designed to assist have similar perceptions of these programs' impact.

Technical issues in the design of these research studies further limit the ability to draw inferences and to generalize findings to other Tribes or populations. Several studies, for example, are limited to a small number of Tribes in selected States. To the extent that there are cultural differences and that health, social, and economic conditions differ across Tribes, the program structure, services provided, and effectiveness of Tribal management may also differ. A few studies (such as those that compared the percentage of Tribal TANF participants that are engaged in work activities to that for State TANF programs) actually incorporated comparison groups in their design. Because most studies did not incorporate a comparison group in their design, it is not possible to determine how persons participating in Tribal programs fare compared to how they would have fared if control over these programs were still vested with the federal or State government.

One reason for the limitation of existing studies and reliance on qualitative techniques to examine the Tribally-managed health and social service programs is the limited availability of data. For instance, the GAO (2002) evaluation of community contracting for health services in Alaska was unable to assess changes in service availability as contracts were switched from regional health organizations to the community because of their relatively recent implementation, the limited scope of services covered under these contracts, and the unavailability of data. Similarly, the evaluation of the TTANF program conducted by the Administration on Children and Families (2002) specifically indicated that because of data limitations "it is too early to come to any firm conclusions about the success of TTANF programs in meeting their negotiated work participation rates." Among the data problems cited was the fact that several Tribes have established agreements with the State to transmit data to DHHS; in several cases the actual transmission of data had not yet occurred. The GAO (2002) survey of TTANF program directors suggested that basic data necessary for Tribes to operate TANF programs, such as estimates of the number of American Indians in the State who receive TANF benefits, are often of poor quality.

The limitations of existing research on process, structure, and impacts of Tribal management of health and social services are due to three major issues: 1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not generalizable; and 3) the lack of adequate and comparable data across Tribally-managed programs and between Tribally-managed programs and federal and State managed programs. In addition, Tribes may be reluctant to cooperate and participate in evaluation studies, either because they may not perceive a value to them or because they perceive that there are risks that, if the programs are found to be successful, the federal government may interpret these findings as an indication that reductions in federal funding or program support can be justified.

The evidence drawn from the review of the literature suggests that qualitative research has been the primary approach to evaluating Tribal management of health and social services programs to date. Quantitative research, however, has been very limited in past studies, due to lack of reliable and comparable data for Tribally-managed programs (and other factors). A primary focus of the Tribal Self-Governance Evaluation Feasibility Study, therefore, will be to review and identify potential sources of data that would be adequate to permit a quantitative evaluation of relevant issues. In addition, involvement of the Technical Working Group and other Tribal representatives in designing the evaluation feasibility study may increase the likelihood that the Feasibility Study will provide useful information and a process that will ensure that any evaluation that is eventually conducted will be useful to Tribes.

VII. References

Bauman, D, Floyd, J, Karshner, B, Siedl, T, Sizemore, J. Indian Tribal Health Systems Governance and Development: Issues and Approaches, Washington, DC: The Henry J. Kaiser Family Foundation, September 1999.

Crow LK. "State Collaboration with Tribal CCDF Programs," Child Care Bulletin, Issue 26, August 23, 2002.

Hillabrandt W, Rhoades MB, Pindus N, Trutko J. The Evaluation of the Tribal Welfare-to-Work Grants Program: Initial Implementation Findings. Princeton, NJ: Mathematica Policy Research, Inc. November 2001.

Kathryn M. Buder Center for American Studies and the Native Nations Institute for Leadership, Management and Policy. Welfare, Work, and American Indians: The Impact of Welfare Reform, A Report to the National Congress of American Indians, November 27, 2001.

Kauffman J. Welfare Reform and American Indian Tribes: Critical Decisions for the Future of Indian Families, Washington DC: The Henry J. Kaiser Family Foundation, 2002.

National Indian Health Board (NIHB). Tribal Perspectives on Indian Self-Determination and Self-Governance in Health Care Management. Denver: National Indian Health Board, 1998.

Noren J, Kindig D, Sprenger A. "Challenges to Native American Health Care", Public Health Reports 1998, 113:22-33.

Pandey S, Brown EF, Shan M, Hicks S, Welch P. State of Welfare Families on Reservations: Progress, Setbacks, and Issues for Reauthorization. St. Louis: Kathryn M. Buder Center for American Studies, September 12, 2001.

Pandey S, Zhan M, Collier-Tenison S, Hui K. How are Families on Reservations Faring Under Welfare Reform? St. Louis: Kathryn M. Buder Center for American Studies, July 20, 2000.

Pandey S, Brown EF, Scheuler-Whitaker L, Gundersen B, Eyrich K, Villareal L. Implementation of the Temporary Assistance for Needy Families (TANF) on American Indian Reservations: Early Evidence from Arizona. St. Louis: Kathryn M. Buder Center for American Studies, October 26, 1999.

United States Department of Health and Human Services, Administration for Children and Families, Temporary Assistance to Needy Families Program (TANF), Fourth Annual Report to Congress, Washington DC: DHHS, April 2002.

United States Department of Health and Human Services, Office of the Inspector General. Tribal Child Care. Washington DC:DHHS, October 1998.

United States General Accounting Office (GAO). Welfare Reform: Tribal TANF Allows Flexibility to Tailor Programs, but Conditions on Reservations Make it Difficult to Move Recipients into Jobs. Washington DC: GAO, July 2002.

United States General Accounting Office (GAO). Indian Self-Determination Contracting: Effects of Individual Community Contracting for Health Services in Alaska. Washington, DC: GAO, June 1998.

United States Department of Health and Human Services, Indian Health Service. Final Report: Methodology for Adjusting IHS Mortality Data for Inconsistent Classification of Race-Ethnicity of American Indians and Alaska Natives Between State Death Certificate and HIS Patient Registration Records. Rockville, MD: November 1996.

United States Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. Tribal Self-Governance Feasibility Study (Draft). Washington, DC: November 5, 2002.

United States Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. Major Statutes Relevant to Self-Governance. (accessed at http://www.aspe.hhs.gov/selfgovernance/ on May 26, 2002.

VIII. Appendix: Literature Search Methods

The literature review and synthesis provides a foundation of information for defining key issues for the design and conduct of the Tribal Self-Governance Evaluation Feasibility Study. In addition, the findings will be shared with the Technical Working Group for review and discussion. Our approach to this task was designed broadly to identify, obtain, and assess published and unpublished research and evaluations of Tribal self-governance/management of health and social services programs, focusing specifically on DHHS programs that have been identified as feasible ones to include in a Tribal Self-Governance Demonstration project.

Based on our preliminary literature review conducted as background for the proposal to OASPE, we anticipated that standard literature search techniques would produce a limited number of published studies of the processes and outcomes associated with Tribal self-governance of federal programs. Consequently, the literature search and review methods used for this report include standard literature search techniques and supplementary activities, including:

• Search of internet websites to identify background papers, issue papers, data sources, projects, and studies that have addressed the relevant issues for this project.
• Telephone interviews with researchers who have been involved in studies of American Indian/Alaska Native health and social services programs, to identify past and ongoing research projects and findings that may be relevant to this study.
• Search of websites of federal government agencies that have responsibilities for health, education, employment, and social services to identify relevant data sources, studies, and initiatives for this study.

The first step in the literature survey was to conduct a thorough search of all published literature through standard literature sources, including:

• Medline
• MedlinePlus: AI/AN Health
• Native Health Research Database
• ERIC

These sources enabled us to identify relevant published literature, from which we compiled a comprehensive bibliography, organized by key topic areas. We then obtained relevant full text and prepared brief abstracts of each publication. As a secondary step, we also searched references cited in each publication to identify additional relevant literature.

Once the published literature bibliography was compiled, we expanded our search through identifying and reviewing websites of national Indian organizations that are concerned with health and social service issues, as well as organizations that are specifically focused on serving and advocating on behalf of AI/AN persons with disabilities. These organizations included:

• National Council of American Indians
• National Indian Health Board
• National Indian Council on Aging
• Association of American Indian Physicians
• National Indian Education Association

In addition, we searched relevant federal government websites(8), including:

• Indian Health Service
• Administration for Native Americans
• Administration on Aging
• Administration for Children and Families
• Centers for Medicare & Medicaid Services
• General Accounting Office

After all the literature was synthesized by topic area, with key findings highlighted, we then reviewed each topic area for completeness and "gaps." The questions addressed in this review included:

1. What do we know with reasonable certainty, based on valid and reliable research?
2. What do the research findings suggest, for which supporting evidence is weaker?
3. What important issues, in this area, have not been addressed by any research?
4. What are the reasons that these issues have not been addressed (e.g. lack of appropriate data)?

This Draft Literature Review includes a summary of findings of this review, limitations and "gaps" in the research and findings, and a discussion of the implications of the findings for the design and conduct of the Tribal Self-Governance Evaluation Feasibility Study.

Notes

1. The information in this section has been drawn from a number of DHHS sources available on the self-governance website (accessed at http://www.aspe.hhs.gov/selfgovernance/).
2. The research questions listed are illustrative examples. The actual set of evaluation issues and research questions will be developed by OASPE/DHHS, with input from the Technical Working Group, consultation with Tribal representatives, and based, in part, on findings from the Tribal Self-Governance Evaluation Feasibility Study.
3. As an example, IHS program staff acknowledge that they do not have a way to make reliable estimates of the number of people who are eligible to use IHS direct service program benefits on specific Reservations, since there are no reliable data on the number of enrolled members of federally recognized Tribes who also live on or near Reservations.
4. Cox, D. and Langwell, K. Expenditures and Sources of Financing of Native American Health. Report prepared for the Henry J. Kaiser Family Foundation, Washington, DC:November 1998.
5. Internal IHS report on 10 year expenditure trends, provided by Cliff Wiggins.
6. www.udallcenter.arizona.edu/publications/welfare_reform/8.html, accessed 2/24/03.
7. The comparison between TTANF and State TANF programs is also tainted because work activities vary for TTANF programs and may include activities such as fishing, hunting, and gathering, that are not counted by State TANF programs.
8. We were not able to conduct a search of the Bureau of Indian Affairs site, because it has been temporarily closed down due to judicial order.