Feasibility Study for the Evaluation of DHHS Programs Operated under Tribal Self-Governance. 6.5 - Comprehensive Evaluation Model – Health Programs

11/30/2003

A comprehensive evaluation of DHHS health programs managed by Tribes under compacts would involve examination of implementation and operational experiences and analysis of the impact of Tribal management on process and outcomes.  As with the illustrative comprehensive evaluation of non-health DHHS programs, Tribes would identify unique program objectives and evaluation of outcomes would examine these unique objectives for each Tribe.

The comprehensive evaluation of DHHS health programs managed under compacts could be conducted as a separate evaluation, as described in Appendix D, or could be conducted as part of a comprehensive evaluation of all DHHS programs managed by Tribes under self-governance.  In the latter case, a subset of Tribes that participate in the new demonstration program and that currently compact for health could be used to evaluate DHHS health programs managed under compacts.

Tribes have been managing their health systems under compacts for a decade, so an evaluation of implementation of these programs and operational experiences would necessarily be retrospective in nature.  The retrospective nature of the evaluation might introduce some biases in the findings, but could provide useful information and insights for DHHS and Tribes.  Because nearly all Tribes and Tribal organizations that compact for health services report data to the IHS Resource and Patient Management System and these data are available for the pre-compact and post-compact period, little primary data collection would be necessary.

However, there are sensitive political issues that would likely affect the feasibility of conducting an evaluation of DHHS health programs operated by Tribes under compacts – particularly if the design involved comparison of compacted health programs with IHS direct service programs.  Even if an alternative comparison strategy was used, that did not compare compacted health programs with direct service programs, it is likely that a prolonged and extensive consultation process between DHHS and the Tribes would be necessary to discuss all aspects of an evaluation of DHHS health programs operated under compacts and it is not certain that any agreement would be reached. 

With respect to the feasibility issues of interest:

  • Willingness of Tribes to Participate.  Tribes that manage health programs under compacts are likely to be reluctant to participate in a comprehensive evaluation of these programs, because they have been managing them for a number of years and because there appear to be political concerns about any potential evaluation of these programs.  This is particularly the case, if the evaluation included examination of financial resources and performance.  Even if the evaluation was limited only to implementation and operational experiences and an evaluation of the extent to which Tribes achieved specific Tribally-set objectives, it is uncertain whether a sufficient number of Tribes would agree to participate.  If only a few Tribes agreed to participate, then the issues of representativeness and usefulness of the evaluation findings would be a concern.
  • Availability of Appropriate Comparisons.  Assuming that the IHS RPMS data are available for participating Tribes and for the past 10 years, pre-post comparisons would be possible.  External comparisons could be based on a selected sample of IHS direct service Tribes or on aggregate IHS data that includes both compact and direct service Tribes.  However, either of those alternative external comparison groups is likely to render the evaluation infeasible due to the political sensitivity of this issue. 
  • Data Availability.  The availability of the RPMS and other IHS data would make it feasible to conduct the comprehensive evaluation of DHHS health programs and little primary data collection would be required.  Some people with whom the project team discussed the study expressed concerns, however, about the quality and completeness of the RPMS data.  It is possible that substantial work would be required to create the evaluation data base to ensure that the quality of the data were sufficient to produce reliable evaluation results.
  • Costs.  Assuming that the only primary data collection was to obtain information on satisfaction and experiences of patients using Tribally-managed and direct service facilities and site visits to 15-25 Tribes, and that the RPMS and other IHS data were available and usable, the evaluation of DHHS health programs managed under compacts could be conducted at a cost of approximately $750,000 to $1,500,000.
  • Trade-offs Between Comprehensiveness and Costs.  The costs of a comprehensive evaluation of DHHS health programs could be less if a decision was made not to collect primary data on patient satisfaction and experiences.  It also could be reduced if a smaller sample was examined for the evaluation (e.g. 10 Tribes rather than 15-25).  If both of these changes were made, the evaluation costs might be reduced to $500,000 to $1,000,000.

In summary, an evaluation of DHHS health programs operated by Tribes under compacts is technically feasible and the cost of such an evaluation would be moderate.  However, there are political considerations that might affect the willingness of Tribes to agree to participate and, thus, might render the evaluation infeasible from a practical standpoint.

View full report

Preview
Download

"draftreport.pdf" (pdf, 169Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®