Tribal Self-Governance Health Care and Social Services Delivery Effectiveness Evaluation Feasibility Study: Draft Evaluation Issues, Questions, and Data Requirements. 2.3.2 Quantitative Methods

03/06/2003

Quantitative methods are necessary to evaluate the outcomes associated with Tribal management of health and social services programs and to understand the factors that contribute to successful programs. Consistent, reliable, and comparable data are necessary to examine research questions on the impact of Tribal management on performance measures and outcomes.

Evaluation that includes quantitative measurement of performance and outcomes requires a carefully structured comparison methodology and appropriate data for a sufficiently long time period to permit accurate attribution of the results to the effects of Tribal management of programs. For Tribally-managed programs that have been operational for a number of years, the most reliable approach to evaluation is:

  • Pre-Post Comparisons. A pre-post comparison of quantitative measures (e.g. non-IHS revenues directed to enhancing or increasing services; number of services provided per user) allows the evaluator to assess the change in the measure that can be attributed to Tribal management. Assessment of the impact, however, also requires that the evaluator be able to 'hold constant' other factors. For instance, to assess the impact of Tribal management on number of people served by a specific program requires that the evaluator also take into account other factors that may affect the number of people who want to use these services. If a Tribe's population of children under age 6 is declining, for example, the number of children served by a Head Start program might decline; but the proportion of all eligible children served by a Tribally-managed Head Start program could increase at the same time. Thus, the ability to do a reliable pre-post comparison that takes into account all relevant factors requires: 1) that the Tribe has operated the program for a number of years, so that a stable management system is in place; 2) that data be available for relevant variables for the period before the Tribe took over management of the program; 3) that data comparable to the pre-Tribal management data be available for the years that the Tribe has managed the program; and 3) that the evaluator is able to identify other factors that may affect the quantitative pre-post comparisons and be able to 'adjust' the results to account for these factors.
  • External Comparisons. When conducting a pre-post comparison evaluation, it is also very important to take into account underlying trends and changes that affect all Tribes and all programs, whether direct service or Tribally-managed. For example, if medical care cost inflation is 10% nationally, then any evaluation of Tribally-managed health services costs must take into account the underlying inflation rate - rather than assuming that real expenditures rose 10% a year for Tribally-managed health programs. Similarly, if all TANF programs experience an increase in caseloads due to economic recession and higher unemployment rates, then this underlying trend must be incorporated into any evaluation of Tribal TANF before the evaluator can assess the true impact of Tribal TANF on number of people served. To 'control' for underlying national trends and factors, it is necessary to be able to obtain appropriate national (or national program) data that can be used to adjust quantitative measures of Tribal management impacts for 'what would have occurred in the absence of Tribal management.'

A comprehensive evaluation requires very careful and rigorous design and appropriate comparisons to ensure that the analysis is capable of identifying real impacts and effects of the program studied. In addition, if there is a quantitative component of the evaluation, the availability, completeness, and quality of the data used for the evaluation is of critical important. Finally, knowledgeable interpretation of the quantitative results are of critical importance for an evaluation to provide accurate and useful information on impacts and effects of a program.

The limitations of existing research on process, structure, and impacts of Tribal management of health and social services are due to three major issues: 1) many of the programs that are currently managed by Tribes have not been in existence for a sufficient time to permit an assessment of the longer-term effects and effectiveness of Tribal management; 2) Tribes are unique in cultural, socioeconomic, and geographic circumstances and, as a result, successful program structures and effectiveness may also be unique and not generalizable; and 3) there is a lack of adequate and comparable data across Tribally managed programs and between Tribally managed programs and federal and State managed programs.

The Evaluation Feasibility Study will primarily address the first and last of these three major barriers to conducting an evaluation of Tribal self-governance and Tribal management of DHHS programs - that is, it will focus on whether Tribes have managed programs for a sufficiently long time period for an accurate assessment of the effects of Tribal management to be conducted and whether adequate and comparable data, over time, are available or could be collected to permit evaluation of Tribally managed programs. However, the feasibility of an appropriate evaluation also must take into account the differences among Tribes and their circumstances that may affect results of any evaluation. To the extent possible, the Evaluation Feasibility Study will also consider this issues and its implications for the feasibility and design of an evaluation.