Submitted to:
Andrew Rock, Task Order Manager
Office of Planning and Evaluation
Department of Health and Human Services
200 Independence Ave., SW, Suite 447-D
Washington, DC 20201
Delivery Order 27
Under Contract No. HHS-100-97-0017
Submitted by:
Westat
1650 Research Blvd.
Rockville, MD 20850
Authors: Janet Sutton, Ph.D. and Kathryn Langwell of Project HOPE Center for Health Affairs prepared this Report. Sreelata Kintala and Valerie Meiners provided research assistance in the literature search and development of the bibliography.
"1. Overview and Objectives
1.1 Overview
The Tribal Self-Governance Evaluation Feasibility Study, being conducted by Westat, and its subcontractors, Project HOPE Center for Health Affairs and Kauffman and Associates, Inc., will provide the Office of the Assistant Secretary for Planning and Evaluation (OASPE) with background information and a detailed review of issues, data availability, and data systems that may affect the extent to which a rigorous and defensible evaluation of Tribal Self-Governance of Indian Health Service and other non-IHS programs can be conducted. While a number of assessments of Tribal self-governance programs have been conducted, these have been primarily qualitative in nature. OASPE is interested in determining the feasibility of conducting an evaluation that examines processes and program changes associated with successful self-governance programs, as well as impacts of Tribal self-governance on outcomes, including access to care, services, quality, costs, financial performance and resources, customer satisfaction, and program stability.
This Draft Literature Review represents one component of the background information that is being assembled to provide a foundation for the development of the evaluation issues, and related data requirements, that will guide that design of the feasibility study. The objectives of this literature review include:
· Identification of existing studies and evaluations of Tribal self-governance and/or Tribal management of health and social service programs;
· Review of the methodologies and data sources used in previous studies, in order to assess both analytic rigor and generalizability of their findings;
· Synthesis of the available evidence and findings from existing studies; and
· Assessment of the limitations of previous studies, data limitations and availability, and areas in which there are few or no existing findings, and the implications of these findings for the Tribal Self-Governance Evaluation Feasibility Study.
In the next section of this Draft Report, the background and definition of Tribal self-governance and Tribally managed programs are reviewed and a list of the relevant health and social services programs for the literature search and review is provided. Then, a description of the methods used to identify relevant literature, both published and unpublished, is provided in Section III. Sections IV and V present a summary and assessment of the literature on Tribal self-governance/management of health programs and of social programs, respectively. Section VI discusses limitations of the existing research and gaps in the literature. The implications of the findings from the literature review for the design and approach to the Tribal Self-Governance Evaluation Feasibility Study are presented in Section VII. Appendix A to the Draft Report briefly describes the literature search methods that were used to identify relevant published and unpublished studies.
2. Background on Tribal SELF-GOVERNANCE/TRIBAL Management of Health and Social Services Programs
2.1 Tribal Self-Governance/Tribal Management of Health and Social Service Programs
Tribes currently manage a number of Department of Health and Human Services programs under several arrangements, including self-governance compacts, self-determination contracts, competitive grants, and demonstration programs. Each of these arrangements may provide differing degrees of autonomy to Tribes and may involve different reporting and compliance requirements.
The authority for Tribal management of federally funded programs was initially provided by Congress under the Indian Self-Determination and Education Assistance Act of 1975 (P.L. 93-638).· One-quarter reported adding mental health services, compared to 14 percent of IHS health directors;
· One-fifth reported that they added alcohol treatment services, compared to 5 percent of IHS health directors who reported adding substance abuse services; and
· Nearly one-quarter reported adding dental services, compared to.10 percent of IHS health directors.
Interestingly, Tribally managed programs were less likely than IHS programs to have added services for diabetes care. Only 12 percent of health directors of Tribally managed programs indicated that diabetes care services had been added, compared to nearly 30 percent of respondents from IHS direct service programs.
Although the literature indicates that Tribes either perceive that contract funding is inadequate or are financially struggling to administer health programs (National Indian Health Board, 1998; Noren et al., 1998; GAO 1998) there is little evidence to indicate how access to services may be affected by financial concerns. In the NIHB survey, 20 percent of respondents from Tribally managed programs indicated that they had to close health facilities; however, less that one-half of these respondents indicated that the closure was related to funding problems. In fact, only 4 percent of Tribes ceased management of health programs altogether. While insufficient budgets were cited as a reason for terminating their participation in contracting/compacting, geographic barriers and regulatory factors were also mentioned as reasons for this decision.
The 1998 GAO study of Tribally managed health services in Alaska also concluded that service availability was generally unaffected when a community takes over health services from a regional health organization (RHO). RHOs are non-profit entities that contract with the IHS to manage and deliver health services to a Tribe. Instead of having the RHO manage their health service delivery, several Alaska Native communities have chosen to directly contract with the IHS. In transitioning to community control, some service disruptions were noted; however, because contracts between the IHS and communities were generally limited to a narrow set of services (typically alcohol abuse and mental health services, health education and non-physician services), the impact on the community was thought to be minimal. Moreover, since contracts are generally program transfers, where the community takes over from the IHS or RHO the management responsibility for existing services, staffing and services are often unchanged. The GAO cautioned that the availability of contract managed health services, which tend to have higher administrative and indirect costs, may be compromised in the future if funding for contract support services is reduced. (Contract support funds refer to the reasonable costs reimbursed by IHS to Tribal communities to cover contract compliance and program management activities.)
3.2 Quality of Care
There is little quantitative evidence on the quality of care rendered by Tribally managed health programs and how patterns of care or outcomes may differ from health programs operated by the IHS. Although the NIHB study that was previously discussed did attempt to gather information on quality of care, quality was subjectively measured. In terms of one standard measure of quality – average wait time – 86 percent of Tribal leader respondents from compacting Tribes indicated that they had noted improvements in wait times over the 3-4 years referenced. In contrast, only 41 percent of Tribal leader respondents from contracting Tribes and 19 percent from the IHS direct service programs indicated that wait time had improved over the past years.
Tribal leaders and health directors were asked about their perceptions of the quality of care that the health systems provides to their Tribe and changes in the quality of care between 1993 and 1996. Representatives of contracting and compacting Tribes were more likely than respondents from the IHS to indicate that the quality of care had improved during this time period. Approximately 94 percent of Tribal leaders and Tribal health director respondents from contracting and compacting Tribes perceived an improvement in quality of care compared to only 62 percent for IHS direct service programs.
These findings, of course, are based on subjective perceptions. To date, no independent quantitative assessment of services provided and of quality of care and outcomes has been conducted.
4. Evidence on Tribal Management of Social Programs
Even though evidence on the impact of self-governance of IHS health programs is limited, it may be possible to gather insight on the effects of Tribal management of services from the literature on Tribal management of social services and non-IHS health programs. One program for which evidence concerning the impact of Tribal management is available is Temporary Assistance to Needy Families (TANF).
The 1996 Personal Responsibility and Work Opportunity Reform Act (PRWORA), which replaced the Aid to Families with Dependent Children (AFDC) with TANF, included provisions that permitted Tribes to operate their own TANF programs. Under the Tribal TANF (TTANF) program, and unlike State TANF programs, Tribes have the flexibility to establish their own work participation goals and to identify the work-related activities that may meet their self-designated work participation goal. Moreover, whereas State TANF participants are eligible to receive cash benefits for a period of up to 60 months, TTANF programs may determine their own time limits. According to the Department of Health and Human Services, Administration for Children and Families (2002), in 2001 a total of 34 Tribal TANF programs, representing 172 Tribes, had been approved.
Despite a national trend toward decreased caseloads, Tribal caseloads have increased or remained the same since the inception of the TTANF program (GAO, 2002). Research to understand the reasons for this growth and the impact of TTANF programs has focused largely on describing the demographic and socio-economic characteristics of persons served by TTANF programs and the economic conditions (e.g., high unemployment rates, lack of skilled labor), social conditions (e.g., lack of child care or employment supports), and the physical infrastructure (e.g., poor roads, limited public transportation, lack of telephones) that may pose barriers to Tribal implementation of these programs or that prevent TTANF programs from achieving their employment goals (Pandey et al., 1999; Pandey et al., 2000; Pandey et al., 2001; GAO 2002.)
Few studies, however, have directly examined the extent to which Tribal TANF programs are achieving their intended objective of promoting independence through employment. The data on work participation rates that has been reported by the Administration for Children and Families (2002) do indicate that TTANF work participation rates average about 37 percent, with approximately one-third of these individuals engaged in unsubsidized employment, one-third engaged in job search activities, and 8 percent engaged in unpaid work. Other program participants were engaged in TANF eligible activities such as subsidized employment, education and vocational training. Because data on the work participation rates of American Indians who participate in State TANF programs were not reported, it is not possible from this study to ascertain how TTANF programs perform relative to State programs. A GAO (2002) study also noted that, in fiscal year 2001, 43 percent of State TANF recipients were involved in work activities (compared to 37 percent of TTANF recipients). Moreover, 60 percent of persons in State TANF programs were in unsubsidized jobs, compared to only one-third of those in Tribal TANF programs.
Insight on the performance of TTANF programs may be gleaned from a series of studies conducted by staff at the Kathryn M. Buder Center for American Indian Studies (Pandey et al. 1999, 2000, 2001). Staff associated with this Center conducted multiple waves of interviews with members of American Indian families, located on three Arizona Reservations, that were currently or had previously received welfare. A total of 350 persons were included in their second round of interviews. Of the Tribal TANF recipients, 15 percent found employment and exited TANF. Although this figure is substantially lower than national estimates of employment of TANF recipients (23 percent), it represents an increase from the previous survey round, when only 11 percent were able to find jobs and exit TANF.Evidence on the effectiveness or impact of Tribal management of health and social services programs is weak and clearly insufficient to draw conclusions concerning the impact or effectiveness of Tribal management, whether under self-governance compacts or under contractual arrangements. Our review of the literature suggests that the information on these issues that may be extracted from the small number of research studies on Tribal self-governance is limited by data and design considerations. Small sample sizes, poor response rates, and the lack of control groups make it difficult to determine the actual effect of these programs or whether these programs may be successfully replicated in other Tribes.
Most of the studies reviewed employed qualitative techniques, such as key informant interviews, which relied on stakeholders’ perceptions to base conclusions about program effectiveness. These qualitative studies offer excellent insight into how various Tribes structure their health and social service programs, the characteristics of Tribal residents participating in these programs, and successes encountered in program implementation. Further, these studies – particularly those that focused on the TTANF programs – effectively highlight how the social and economic conditions on the Reservation, such as the high rate of poverty, high unemployment rates, and the lack of an economic base -- may pose substantial barriers to achieving the intended goals of these programs. These studies do not, however, provide reliable quantitative evidence on the extent to which and how Tribally managed health and social service programs have operated to better meet the needs of their members. Moreover, because the small number of studies that directly examined issues of access and quality based their conclusions on interviews or surveys of Tribal leaders or program directors, it is unclear whether the Tribal members that these programs are designed to assist have similar perceptions of these programs’ impact.
One reason for the limitation of existing studies and reliance on qualitative techniques to examine the Tribally managed health and social service programs is the limited availability of data. For instance, the GAO (2002) evaluation of community contracting for health services in Alaska was unable to assess changes in service availability as contracts were switched from regional health organizations to the community because of their relatively recent implementation, the limited scope of services covered under these contracts, and the unavailability of data. Similarly, the evaluation of the TTANF program conducted by the Administration on Children and Families (2002) specifically indicated that because of data limitations “it is too early to come to any firm conclusions about the success of TTANF programs in meeting their negotiated work participation rates.” Among the data problems cited was the fact that several Tribes have established agreements with the State to transmit data to DHHS; in several cases the actual transmission of data had not yet occurred. The GAO (2002) survey of TTANF program directors suggested that basic data necessary for Tribes to operate TANF programs, such as estimates of the number of American Indians in the State who receive TANF benefits, are often of poor quality.
The limitations of existing research on process, structure, and impacts of Tribal management of health and social services are due to three major issues: 1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not generalizable; and 3) the lack of adequate and comparable data across Tribally managed programs and between Tribally managed programs and federal and State managed programs.
Of these three issues, the greatest challenge for the conduct of rigorous quantitative evaluations is the lack of adequate and comparable data. This lack of data is due to a number of factors. First, Tribes who elect self-governance of IHS health programs are not required to report specific and comparable data to IHS that would permit evaluation of outcomes, relative to outcomes of IHS direct service provision. Instead, each self-governance Tribe negotiates with IHS to identify specific measures that are relevant and unique to its community. In fact, one of the positive benefits of self-governance of IHS health programs is the low burden of reporting requirements. For other HHS health and social services programs that are managed by Tribes under contracts or grants, reporting requirements may be limited and may be different from reporting requirements for States and federal program offices, and the data submitted may not be accumulated and maintained in a database that is adequate for research purposes.
Second, the American Indian/Alaska Native population represents only about one percent of the U.S. population. Even very large sample national surveys, that provide base data used for many evaluations, seldom obtain sufficient observations of the AI/AN population to permit reliable estimates of socioeconomic, health status, and other characteristics of this population at the sub-State or Reservation level. As a result, there is often no baseline data that could be used to assess the size and characteristics of the potential eligible population to be served by the Tribally managed program or the services that the affected population received prior to the implementation of Tribal management.
Third, American Indians and Alaska Natives may not be accurately identified in many national datasets. As one example, the Indian Health Service conducted a study of the accuracy of AI/AN race coding on State Death Certificates and found, on average, there was an 11 percent miscoding of AI/AN’s as other races and that, in some States, the proportion miscoded was as high as 47 percent (November 1996).
Finally, for many research purposes, the issues of membership in a federally recognized Tribe and geographic location on or near a Reservation are often critical ones. Indian people may be enrolled members of a federally recognized Tribe, members of a State recognized Tribe, or of AI/AN heritage but not an enrolled member of any Tribe. They may live on or near a Reservation or in areas far from a Reservation. Eligibility for benefits under Tribally managed health and social services programs may be restricted to enrolled members of federally recognized Tribes, or to enrolled members of a specific federally recognized Tribe. At this time, however, there are no reliable data that would permit desaggregation of the AI/AN population by Tribal membership and geographic residence. [3]
The lack of consistent and comparable program data, the relatively small AI/AN population that makes most national survey databases inadequate for study of this population, the problem of inadequate identification of AI/AN race that calls into question data that are available, and the complexity of defining the potential eligible program population are all major obstacles to designing and conducting a reliable quantitative evaluation of Tribal management of health and social services programs.
6. Implications for the Tribal Self-governance Evaluation Feasibility Study
A comprehensive and rigorous evaluation of processes, structure, and outcomes associated with Tribal management of health and social services programs would use both qualitative and quantitative analyses to address the issues of importance to understanding the benefits of Tribal management and the factors that contribute to the success of Tribal self-governance/management.
Qualitative methods – key informant interviews, site visits, surveys of perceptions – can provide useful insights and understanding of research questions such as: [4]
· What are the goals of Tribes that manage health and social services programs?
· How are programs structured differently under Tribal management?
· What changes are made in the programs, and why were these changes made?
· What problems were encountered in establishing Tribally managed programs? How were these problems resolved?
· How are community members involved in defining priorities and providing input to guide programs?
· What are the recommendations of Tribal leaders, Tribal program managers, and Tribal program staff that could help them improve services and manage more effectively?
Quantitative methods are necessary to evaluate the outcomes associated with Tribal management of health and social services programs and to understand the factors to contribute to successful programs. Consistent, reliable, and comparable data are necessary to examine research questions on the impact of Tribal management on measurable performance outcomes, such as:
· How do the numbers and types of services offered change?
· How many people use services, by type? Are services routinely available or is there a significant delay in access or a waiting list?
· What are the outcomes achieved by the program (e.g., percentage of clients receiving preventive health services, increased employment rates)?
· Is the program able to recruit and retain appropriate professional staff to avoid vacancies?
· Is the financial management of the program stable and adequate? Are additional sources of revenues obtained to supplement the base allocation from the federal agency? What are these additional sources of revenue and how much additional funding comes from each?
The evidence drawn from the review of the literature suggests that qualitative research has been the primary approach to evaluating Tribal management of health and social services programs to date. Quantitative research, however, has been very limited in past studies, due to lack of reliable and comparable data for Tribally managed programs. A primary focus of the Tribal Self-Governance Evaluation Feasibility Study, therefore, will be to review and identify potential sources of data that would be adequate to permit a quantitative evaluation of relevant issues.
References
Bauman, D, Floyd, J, Karshner, B, Siedl, T, Sizemore, J. Indian Tribal Health Systems Governance and Development: Issues and Approaches, Washington, DC: The Henry J. Kaiser Family Foundation, September 1999.
Crow LK. “State Collaboration with Tribal CCDF Programs,” Child Care Bulletin, Issue 26, August 23, 2002.
Hillabrandt W, Rhoades MB, Pindus N, Trutko J. The Evaluation of the Tribal Welfare-to-Work Grants Program: Initial Implementation Findings. Princeton, NJ: Mathematica Policy Research, Inc. November 2001.
Kathryn M. Buder Center for American Studies and the Native Nations Institute for Leadership, Management and Policy. Welfare, Work, and American Indians: The Impact of Welfare Reform, A Report to the National Congress of American Indians, November 27, 2001.
Kauffman J. Welfare Reform and American Indian Tribes: Critical Decisions for the Future of Indian Families, Washington DC: The Henry J. Kaiser Family Foundation, 2002.
National Indian Health Board (NIHB). Tribal Perspectives on Indian Self-Determination and Self-Governance in Health Care Management. Denver: National Indian Health Board, 1998.
Noren J, Kindig D, Sprenger A. “Challenges to Native American Health Care”, Public Health Reports 1998, 113:22-33.
Pandey S, Brown EF, Shan M, Hicks S, Welch P. State of Welfare Families on Reservations: Progress, Setbacks, and Issues for Reauthorization. St. Louis: Kathryn M. Buder Center for American Studies, September 12, 2001.
Pandey S, Zhan M, Collier-Tenison S, Hui K. How are Families on Reservations Faring Under Welfare Reform? St. Louis: Kathryn M. Buder Center for American Studies, July 20, 2000.
Pandey S, Brown EF, Scheuler-Whitaker L, Gundersen B, Eyrich K, Villareal L. Implementation of the Temporary Assistance for Needy Families (TANF) on American Indian Reservations: Early Evidence from Arizona. St. Louis: Kathryn M. Buder Center for American Studies, October 26, 1999.
United States Department of Health and Human Services, Administration for Children and Families, Temporary Assistance to Needy Families Program (TANF), Fourth Annual Report to Congress, Washington DC: DHHS, April 2002.
United States Department of Health and Human Services, Office of the Inspector General. Tribal Child Care. Washington DC:DHHS, October 1998.
United States General Accounting Office (GAO). Welfare Reform: Tribal TANF Allows Flexibility to Tailor Programs, but Conditions on Reservations Make it Difficult to Move Recipients into Jobs. Washington DC: GAO, July 2002.
United States General Accounting Office (GAO). Indian Self-Determination Contracting: Effects of Individual Community Contracting for Health Services in Alaska. Washington, DC: GAO, June 1998.
United States Department of Health and Human Services, Indian Health Service. Final Report: Methodology for Adjusting IHS Mortality Data for Inconsistent Classification of Race-Ethnicity of American Indians and Alaska Natives Between State Death Certificate and HIS Patient Registration Records. Rockville, MD: November 1996.
United States Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. Tribal Self-Governance Feasibility Study (Draft). Washington, DC: November 5, 2002.
United States Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. Major Statutes Relevant to Self-Governance. (accessed at http://www.aspe.hhs.gov/selfgovernance/ on May 26, 2002.
Appendix: Literature Search Methods
The literature review and synthesis provides a foundation of information for defining key issues for the design and conduct of the Tribal Self-Governance Evaluation Feasibility Study. In addition, the findings will be shared with the Technical Working Group for review and discussion. Our approach to this task was designed broadly to identify, obtain, and assess published and unpublished research and evaluations of Tribal self-governance/management of health and social services programs, focusing specifically on DHHS programs that have been identified as feasible ones to include in a Tribal Self-Governance Demonstration project.
Based on our preliminary literature review conducted as background for the proposal to OASPE, we anticipated that standard literature search techniques would produce a limited number of published studies of the processes and outcomes associated with Tribal self-governance of federal programs. Consequently, the literature search and review methods used for this report include standard literature search techniques and supplementary activities, including:
· Search of internet websites to identify background papers, issue papers, data sources, projects, and studies that have addressed the relevant issues for this project.
· Telephone interviews with researchers who have been involved in studies of American Indian/Alaska Native health and social services programs, to identify past and ongoing research projects and findings that may be relevant to this study.
· Search of websites of federal government agencies that have responsibilities for health, education, employment, and social services to identify relevant data sources, studies, and initiatives for this study.
The first step in the literature survey was to conduct a thorough search of all published literature through standard literature sources, including:
· Medline
· MedlinePlus: AI/AN Health
· Native Health Research Database
· ERIC
These sources enabled us to identify relevant published literature, from which we compiled a comprehensive bibliography, organized by key topic areas. We then obtained relevant full text and prepared brief abstracts of each publication. As a secondary step, we also searched references cited in each publication to identify additional relevant literature.
Once the published literature bibliography was compiled, we expanded our search through identifying and reviewing websites of national Indian organizations that are concerned with health and social service issues, as well as organizations that are specifically focused on serving and advocating on behalf of AI/AN persons with disabilities. These organizations included:
· National Council of American Indians
· National Indian Health Board
· National Indian Council on Aging
· Association of American Indian Physicians
· National Indian Education Association
In addition, we searched relevant federal government websites [5] , including:
· Indian Health Service
· Administration for Native Americans
· Administration on Aging
· Administration for Children and Families
· Centers for Medicare & Medicaid Services
· General Accounting Office
After all the literature was synthesized by topic area, with key findings highlighted, we then reviewed each topic area for completeness and “gaps.” The questions addressed in this review included:
1. What do we know with reasonable certainty, based on valid and reliable research?
2. What do the research findings suggest, for which supporting evidence is weaker?
3. What important issues, in this area, have not been addressed by any research?
4. What are the reasons that these issues have not been addressed (e.g. lack of appropriate data)?
This Draft Literature Review includes a summary of findings of this review, limitations and “gaps” in the research and findings, and a discussion of the implications of the findings for the design and conduct of the Tribal Self-Governance Evaluation Feasibility Study.