Submitted to:
Andrew Rock, Task Order Manager
Office of Planning and Evaluation
Department of Health and Human Services
200 Independence Ave., SW, Suite 447-D
Washington, DC 20201
Delivery Order 27
Under Contract No. HHS-100-97-0017
Submitted by:
Westat
1650 Research Blvd.
Rockville, MD 20850
1. Overview and Objectives
1.1 Overview
The Tribal Self-Governance Evaluation Feasibility Study, being conducted by Westat, Project HOPE Center for Health Affairs, and Kauffman and Associates, Inc., will provide the Office of the Assistant Secretary for Planning and Evaluation with in-depth background information and detailed review of issues and data systems that may affect the extent to which a rigorous and defensible evaluation of Tribally managed health and social services programs of the Indian Health Service and other non-IHS programs can be conducted. While a number of assessments of Tribally managed services programs have been conducted, these have been primarily qualitative in nature. OASPE is interested in determining the feasibility of conducting an evaluation that examines both processes associated with successful Tribal management of health and social services programs and impacts of Tribal management on outcomes, including access to care, services, quality, costs, financial performance and resources, customer satisfaction, and program stability.
During the initial four months of this project, a substantial amount of information was assembled by the project team, as background for understanding and laying the groundwork for developing the evaluation issues, research questions, and data requirements for the Evaluation Feasibility Study. These activities included:
- Development of a Draft Work Plan for the Evaluation Feasibility Study, including a set of preliminary evaluation issues, questions, and data requirements;
- Review of the research and other literature on Tribal self-governance and preparation of the Draft Literature Review;
- Preparation of a Draft Report on Legislative History and Development of Tribal Self-Governance and Contracting;
- Obtaining data from IHS and other DHHS agencies on specific programs that Tribes are currently managing under self-governance compacts and under contracts and grants, and a Draft Tribal Matrix Report was prepared that indicated specific programs operated by each federally-recognized Tribe;
- Preparation of a Draft Communications Strategy Report, addressing plans and mechanisms for communicating information, throughout the project, to the Evaluation Feasibility Study Technical Working Group, to all Tribes, and other interested parties; and
- Compiling data of socioeconomic, demographic, and other data for each federally-recognized Tribe and preparing a set of tables presenting this information for each Tribe.
In addition, the project team and the Task Order Managers worked jointly to identify and recruit 10 individuals to serve on the Technical Working Group that will advise, guide, and review the process and information through which the Tribal Self-Governance Evaluation Feasibility Study is developed and implemented.
The first meeting of the Technical Working Group was held in Washington, DC on February 3, 4, and 5, 2003. The Technical Working Group provided background, review, and guidance on the project goals. They also provided detailed comments on each of the draft deliverables that had been provided to them in advance of the meeting.
1.2 Objectives of This Draft Report
The Technical Working Group spent a substantial amount of time reviewing and providing guidance on the evaluation objectives, appropriate evaluation issues and research questions, and the data that could be sought from Tribes to support an evaluation. This Draft Report on Evaluation Issues, Questions, and Data Requirements has been developed to clearly incorporate the TWG's input into the Evaluation Feasibility Study design and to build on that input to develop an approach that reflects the guidance and priorities expressed by TWG members.
In the next section of this Draft Report, an overview of the objectives of the Tribal Self-Governance Evaluation Feasibility Study is provided and the implications of those objectives for the design of the Study are discussed. In Section 3, a number of potential evaluation issues are discussed and associated research questions and the data that would be required to address those questions are presented.
2. the Tribal Self-governance Evaluation Feasibility Study
2.1 Background and Objectives of the Study
In the Self-Governance Amendments of 2000 (P.L. 106-260), Congress re-affirmed its commitment to Tribal self-governance. In the Preamble to the Act, the Congress defined the goal of self-governance as "to permit an orderly transition from Federal domination of programs and services to provide Indian Tribes with meaningful authority, control, funding, and discretion to plan, conduct, redesign, and administer programs, services, functions, and activities (or portions thereof) that meet the needs of individual Tribal communities."
The Act established Tribal Self-Governance of Indian Health Service programs on a permanent basis. In addition, the Congress directed the Secretary of DHHS to "conduct a study to determine the feasibility of a Tribal self-governance demonstration project for appropriate programs, services, functions, and activity (or portions thereof) of the agency [HHS]." The Office of the Assistant Secretary for Planning and Evaluation conducted the Tribal Self-Governance Demonstration Feasibility Study in 2001-2001. The Final Report on the Study, released November 5, 2002, identified 11 DHHS programs as "feasible for inclusion in a Tribal self-governance demonstration project". These 11 programs are:
- Administration on Aging
- Grants for Native Americans
- Administration for Children and Families
- Tribal Temporary Assistance for Needy Families
- Low Income Home Energy Assistance
- Community Services Block Grant
- Child Care and Development Fund
- Native Employment Works
- Head Start
- Child Welfare Services
- Promoting Safe and Stable Families
- Family Violence Prevention: Grants for Battered Women's Shelters
- Substance Abuse and Mental Health Services Administration
- Targeted Capacity Expansion
There are Tribes currently managing each of these DHHS programs that are recommended for inclusion in a Tribal Self-Governance Demonstration program, under contractual arrangements or grant awards. A Self-Governance Demonstration program, as detailed in the Final Report, could permit a simpler, multiple-program application process and simpler and consolidated reporting requirements. Importantly, the Demonstration program could provide "Tribes with the flexibility to change programs and reallocate funds among programs" (p.19) to better address specific Tribal community priorities.
Initiation of a DHHS Tribal Self-Governance Demonstration requires Congressional action prior to implementation. With the prospect that Congress may act to authorize such a demonstration, DHHS has identified a need to address the absence of conclusive quantitative evaluation to document the successes and outcomes of Tribal management of health and social services programs. DHHS contracted with Westat to conduct the Tribal Self-Governance Evaluation Feasibility Study to provide background information and to assess the feasibility of conducting a rigorous and defensible evaluation of Tribal management of health and social services programs under self-governance compacts and self-determination contracts.
2.2 Overview of Evaluation Feasibility Study Goals and Products
The Tribal Self-Governance Evaluation Feasibility Study is intended to provide information to DHHS and to Tribes on several questions:
- What evaluation issues and research questions are important to address in an evaluation?
- What measures - both qualitative and quantitative - are appropriate to use to address each of the selected evaluation issues and research questions?
- Are there data available, within DHHS and from selected Tribes, that would permit an evaluation to be conducted, using a rigorous methodology that would be likely to produce reliable results?
- What are the evaluation cost implications of alternative feasible strategies (including sample size cost implications)?
The product of this Evaluation Feasibility Study is expected to be a Final Report that includes: 1) a set of evaluation issues and research questions that would be feasible to address in an evaluation, based on assessment of current data availability; 2) additional data collection that would be required to address other evaluation issues and research questions that cannot be examined because data are not currently available; 3) evaluation issues and research questions that cannot be addressed at all, due to lack of data availability and the difficulties or impossibility of collecting appropriate additional data; 4) limitations and considerations that may affect the feasibility of conducting an evaluation of Tribal self-governance; and 5) estimates of relative costs of alternative evaluation options.
It is important to note that this project will not produce a comprehensive evaluation design and methodology. Any evaluation that is considered will be developed through a consultation process between DHHS and the Tribes, at some point after the completion of this Evaluation Feasibility Study. Results of this Study are intended only to provide information on feasible options for an evaluation and considerations that can be used by DHHS and the Tribes as part of their consultation about the type of evaluation that could be conducted and the range of issues that could be addressed.
2.3 Methodological Issues Guiding the Evaluation Feasibility Study
A comprehensive and rigorous evaluation of processes, structure, and outcomes associated with Tribal management of health and social services programs would use both qualitative and quantitative analyses to address the issues of importance to understanding the benefits of Tribal management and the factors that contribute to the success of Tribal self-governance/management.
2.3.1 Qualitative Methods
Qualitative methods are key informant interviews, site visits, surveys of perceptions that can provide useful insights and understanding of research questions that focus on decisionmaking, management approaches, implementation and operations, and other process issues that cannot be easily quantified. These issues are important to understand the factors that influence and determine Tribal decisions for self-governance or contracting, goals of self-governance/contracting, and the ways that programs are implemented and operated to achieve those goals. In addition, qualitative research is important to obtain information that assists the evaluation team to understand and appropriately interpret quantitative findings of the evaluation..
In support of qualitative and process analyses, it is also important to identify and review historical documentation of implementation activities and records of decision making related to the qualitative outcomes that are identified.
2.3.2 Quantitative Methods
Quantitative methods are necessary to evaluate the outcomes associated with Tribal management of health and social services programs and to understand the factors that contribute to successful programs. Consistent, reliable, and comparable data are necessary to examine research questions on the impact of Tribal management on performance measures and outcomes.
Evaluation that includes quantitative measurement of performance and outcomes requires a carefully structured comparison methodology and appropriate data for a sufficiently long time period to permit accurate attribution of the results to the effects of Tribal management of programs. For Tribally-managed programs that have been operational for a number of years, the most reliable approach to evaluation is:
- Pre-Post Comparisons. A pre-post comparison of quantitative measures (e.g. non-IHS revenues directed to enhancing or increasing services; number of services provided per user) allows the evaluator to assess the change in the measure that can be attributed to Tribal management. Assessment of the impact, however, also requires that the evaluator be able to 'hold constant' other factors. For instance, to assess the impact of Tribal management on number of people served by a specific program requires that the evaluator also take into account other factors that may affect the number of people who want to use these services. If a Tribe's population of children under age 6 is declining, for example, the number of children served by a Head Start program might decline; but the proportion of all eligible children served by a Tribally-managed Head Start program could increase at the same time. Thus, the ability to do a reliable pre-post comparison that takes into account all relevant factors requires: 1) that the Tribe has operated the program for a number of years, so that a stable management system is in place; 2) that data be available for relevant variables for the period before the Tribe took over management of the program; 3) that data comparable to the pre-Tribal management data be available for the years that the Tribe has managed the program; and 3) that the evaluator is able to identify other factors that may affect the quantitative pre-post comparisons and be able to 'adjust' the results to account for these factors.
- External Comparisons. When conducting a pre-post comparison evaluation, it is also very important to take into account underlying trends and changes that affect all Tribes and all programs, whether direct service or Tribally-managed. For example, if medical care cost inflation is 10% nationally, then any evaluation of Tribally-managed health services costs must take into account the underlying inflation rate - rather than assuming that real expenditures rose 10% a year for Tribally-managed health programs. Similarly, if all TANF programs experience an increase in caseloads due to economic recession and higher unemployment rates, then this underlying trend must be incorporated into any evaluation of Tribal TANF before the evaluator can assess the true impact of Tribal TANF on number of people served. To 'control' for underlying national trends and factors, it is necessary to be able to obtain appropriate national (or national program) data that can be used to adjust quantitative measures of Tribal management impacts for 'what would have occurred in the absence of Tribal management.'
A comprehensive evaluation requires very careful and rigorous design and appropriate comparisons to ensure that the analysis is capable of identifying real impacts and effects of the program studied. In addition, if there is a quantitative component of the evaluation, the availability, completeness, and quality of the data used for the evaluation is of critical important. Finally, knowledgeable interpretation of the quantitative results are of critical importance for an evaluation to provide accurate and useful information on impacts and effects of a program.
The limitations of existing research on process, structure, and impacts of Tribal management of health and social services are due to three major issues: 1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not generalizable; and 3) there is a lack of adequate and comparable data across Tribally managed programs and between Tribally managed programs and federal and State managed programs.
The Evaluation Feasibility Study will primarily address the first and last of these three major barriers to conducting an evaluation of Tribal self-governance and Tribal management of DHHS programs - that is, it will focus on whether Tribes have managed programs for a sufficiently long time period for an accurate assessment of the effects of Tribal management to be conducted and whether adequate and comparable data, over time, are available or could be collected to permit evaluation of Tribally managed programs. However, the feasibility of an appropriate evaluation also must take into account the differences among Tribes and their circumstances that may affect results of any evaluation. To the extent possible, the Evaluation Feasibility Study will also consider this issues and its implications for the feasibility and design of an evaluation.
3. Evaluation Issues, Questions, and Data Requirements
3.1 Evaluation Issues
A set of preliminary evaluation issues, research questions, and associated data requirements was prepared for review and discussion with the Technical Working Group. The TWG provided guidance on the general structure and issues for an evaluation of Tribal Self-Governance, as well as specific comments and suggestions for evaluation areas, research questions, and data search methods. General guidance for the project from the TWG included:
- It is very important to examine, broadly, the impacts of Tribal self-governance on Tribe's capacities for management, economic development, and employment. The TWG stressed that self-governance empowers Tribes to develop internal capabilities and economies;
- The management processes that emerge to lead self-governance programs, how they are structured and evolve over time, should be included in an evaluation (that is, "what did the Tribe do and how did they do it?");
- The evaluation should look at the totality of Tribal self-governance and Tribal management of programs under contracts and grants, not just one program;
- Because each Tribe is unique, the evaluation feasibility study should be designed flexibly and should allow for use of measures of performance that vary across Tribes being studied. For example, since Tribes may have different priorities for health programs; the evaluation should be capable of assessing the factors that affect Tribes' priorities and of measuring the extent to which the Tribes are able to address individual priorities;
- The evaluation - and the evaluation feasibility study - should include both qualitative and quantitative components. Quantitative measures alone will not provide the information or accurate reflection of the 'success' of self-governance; and
- Most health status outcomes are not appropriate to include in an evaluation of self-governance of IHS health programs. Self-governance has not been operating over a sufficiently long time period for impacts on health and health status to be observed. There are also many other factors that affect health outcomes (e.g. poverty, education, history of inadequate services) and it would be very difficult to fully account for these factors. Health process outcomes, however, may be feasible to include (e.g. proportion of people over 50 who receive influenza immunizations; proportion of people with diabetes with annual foot examinations).
In addition, the TWG reviewed the preliminary set of evaluation issues and provided input and guidance on appropriate areas for examination. Based on the TWG input and the project team's understanding of OASPE's goals for the Evaluation Feasibility Study, the set of potential evaluation issues for examination in the feasibility study include:
- Qualitative Issues: Overall Effects, Management, Implementation and Operations, Community Involvement
- Self-Governance as a mechanism for strengthening Tribes' capacity for management, economic development, employment;
- Factors that affect the decision for self-governance/Tribal management of programs;
- Processes for establishing community priorities for programs;
- Management structure and stability;
- Implementation processes, problems, solutions;
- Operational processes, problems, solutions;
- Involvement of Tribal leadership and Tribal members in program development;
- Perceptions of Tribal leadership and Tribal members on benefits of Tribal self-governance/management of programs and success in meeting specific program priorities and goals; and
- Effective strategies for augmenting and re-allocating financial resources to support specific programs and across all Tribally managed programs.
- Quantitative Issues: Services Provided, Access, Staffing and Personnel Stability, Financing, Quality, Satisfaction
- For specific programs, extent to which the Tribe was successful in achieving its individual goals for the program (e.g. for self-governance of health programs, if the Tribe's priorities were to increase the proportion of women receiving prenatal care during the initial trimester, did this proportion increase?);
- For specific programs, extent to which the type and quantity of specific types of services changed under Tribal management;
- For specific programs, extent to which access to (i.e. number of people served) and use of services (i.e. number of services provided, overall and per user) changed;
- For specific programs, extent to which the number and types of service providers changed under Tribal management and extent to which retention of service providers changed;
- Across Tribes, extent to which there are common process of care measures that demonstrate improvements in services (e.g. for Tribal self-governance of health programs, do more elderly people receive immunization for influenza?;
- For specific programs and across all programs, extent to which Tribes re-allocate resources, obtain supplemental financial support for programs, and become more efficient in using existing financial resources; and
- For specific programs and across programs, extent to which Tribal members served by the program are satisfied with access and service provided by Tribally managed programs.
For the qualitative issues, the Evaluation Feasibility Study team will review the extent to which written documentation of management decisions, implementation, operations, and periodic reporting are available. In addition, the study team will determine the extent to which there are knowledgeable individuals (e.g. Tribal leaders, program directors) who have been associated with self-governance and Tribally-managed programs for a sufficiently long time period that they can provide the evaluation with the history, perspectives, and details of decision-making, priorities, and experiences over the establishment and operation of Tribally-managed programs.
For the quantitative issues, the Evaluation Feasibility Study team will develop a detailed data review protocol and will review, at each site (and possibly submit the protocol to additional Tribes for written responses), the availability of data to address each evaluation issue.
NOTE: THIS DRAFT REPORT WILL BE EXPANDED TO INCLUDE A DETAILED SET OF RESEARCH QUESTIONS AND THE SPECIFIC DATA REQUIREMENTS ASSOCIATED WITH EACH RESEARCH QUESTION, AFTER THE TECHNICAL WORKING GROUP COMPLETES ITS REVIEW OF THE ABOVE EVALUATION ISSUES AND PROVIDES COMMENTS AND ADDITIONAL GUIDANCE.