Feasibility Study for the Evaluation of DHHS Programs That Are or May Be Operated Under Tribal Self-Governance. Feasibility of Evaluating Indian Health Service Programs Operated by Tribes Under Self-Governance


The Scope of Work for the contract to conduct the Tribal Self-Governance Evaluation Feasibility Study included requirements to assess the feasibility of conducting an evaluation of Indian Health Service programs operated by Tribes under Self-Governance compacts, as well as the feasibility of an evaluation of DHHS programs that may be operated by Tribes under a potential Self-Governance demonstration.  Early in the study, the Technical Working Group for the project indicated that an evaluation of IHS programs operated under Self-Governance was fundamentally different from an evaluation of a new demonstration program.  The TWG emphasized that Self-Governance of health programs is not a demonstration program and Tribes have been operating IHS programs under compacts for over a decade.  Any evaluation that would be conducted would be retrospective in nature and subject to more limitations than would evaluation of a new demonstration.

In addition, the extensive experience with Self-Governance of IHS programs means that Tribes have had many years to develop management and operational systems and are confident that these programs are successful and effective.  The Indian Health Service also has monitored and worked with current compacted Tribes over this period and the program is well-established.

Because nearly all Tribes report data to the Resource and Patient Management System (RPMS) maintained by the IHS, there are extensive data available to conduct a quantitative evaluation of processes and outcomes of Self-Governance. The Indian Health Service Resource and Patient Management System database is an IHS-wide system designed to provide detailed and comprehensive clinical information for assessment and management of IHS performance.  It has several components for reporting detailed information on patient characteristics, diagnoses, and specific services provided.  The data included on the Ambulatory Patient Care System component are collected and entered daily, one record per ambulatory medical visit, and includes information on diagnosis, treatments, and specific examinations and tests performed and their results.  In addition, results of special medical record audits are entered into the database.  The RPMS has the capability to produce special reports, by IHS Region, Reservation, Service Unit, and by individual physician.  A pre-post comparison strategy and/or an external comparison strategy based on direct service IHS facilities would be possible, assuming reliable and consistent RPMS data were available for the time frame that Self-Governance has been in place.  However, the Indian Health Service has not conducted such an evaluation.

In this Appendix, we present the background information and evaluation feasibility findings for Indian Health Service programs operated under Self-Governance compacts.  Information on which the evaluation feasibility study was based includes a review of IHS RPMS data capabilities and site visit findings.  Results of these two activities are described below.  Then, illustrative models for the evaluation of IHS program Self-Governance are described and reviewed for feasibility.

Indian Health Service Data Review

Several IHS staff was interviewed about data available through the Resource and Patient Management System (RPMS).  There were two primary foci in these discussions:  patient-level data and administrative and personnel-related data.  The RPMS is an integrated software system for management of clinical and administrative data in IHS and Tribally operated healthcare facilities.  It is composed of several different data collection components.  The Patient Care Component (PCC) comprises data collected at the patient level regarding all care received through the service units and includes a number of client characteristics.  Among the data elements that were investigated and found to be available in some form from the RPMS, and particularly the PCC, from 1998 forward are those listed in the Table F1 below.

Table F.1:  Selected data available through the Patient Care Component of the RPMS

Unit of Measurement

Data Available

At the Service Unit (SU) level

Number of patients provided services in SU, by age and gender


Number and type of Contract Health Services provided by quarter of the fiscal year


Percent of patients with Medicare coverage


Percent of patients with Medicaid coverage


Percent of patients with SCHIP coverage


Percent of patients with Private Health Insurance coverage


Number of hospital admissions


Number of hospital days


Number of primary care visits


Number of specialist physician visits


Number of dental visits


Number of prescriptions filled


Percent children under age 5 immunized


Percent of patients aged 50+ receiving influenza immunizations


Percent of women over 18 with annual Pap smears


Percent of pregnant women obtaining prenatal care in first trimester


Percent of adults screened for diabetes


Percent of patients diagnosed with breast cancer surviving 5 years


Percent of patients diagnosed with cervical cancer surviving 5 years


Percent of births that are low-weight or premature


Percent of births that are high-weight


Percent of deaths attributable to diabetes


For each Service Unit, for all patients with diabetes, three years:

Percent of patients seeing physician at least once in 3 months


Percent of patients receiving HbA1c testing once in 3 months


Percent of patients receiving dilated eye exam annually


Percent of people with diabetes who have diabetic retinopathy


Percent of people with diabetes who have had amputation

The availability of administrative and personnel-related data elements was also investigated.  These elements would include staffing information, information on staff credentials, staff turnover, pharmacy information, and payment information.  Some of this information are likely available through other components of the RPMS (e.g., accounts payable, contract health, staff credentials), if these components are in use by the Tribal entities of interest. 

Site Visit Findings: Availability, Accessibility, and Quality of Data on Self-Governance of IHS Health Programs

Four of the six Tribes visited have IHS compacts and two of the Tribes have 638 contracts.  The Tribes have managed components of health care for a minimum of four years. Each of the sites indicated that individuals knowledgeable about the Tribe’s experience in Self-Governance or management of health care are available and could provide historical background and other information for an evaluation.  

Accounting and personnel data concerning health programs were available at all sites.  Detailed accounting data were available at all sites including cost information by cost component (e.g., administrative costs, personnel costs, and other) and funding allocations by source over time.  Personnel data including staff turnover information are available at all six sites but are not generally available electronically.  This information would, in most cases, have to be recreated by knowledgeable managers.

All six Tribes use the RPMS system for collecting patient-care data. A few Tribes were also using RPMS data for third party billing or looking into using RPMS for third party billing activities.  Follow-up information gathering with the Indian Health Service revealed that most Tribes/Tribal organizations with Self-Governance compacts (78 of 81 compacts) do submit data to the RPMS.

However, a few of those interviewed expressed concern over the quality of the data collected through the RPMS.  These Tribal interviewees suggested that, without intensive effort at the facility-level to enhance quality, the data were not extremely useful.  One site had enhanced the RPMS data by training staff members in its use, collecting additional outcome data, and conducting a separate patient satisfaction survey.

Illustrative Evaluation Models for Indian Health Service Programs Operated Under Self-Governance Compacts

Illustrative Comprehensive Evaluation Model – IHS Health Programs

A comprehensive evaluation of Indian Health Service programs managed by Tribes under compacts would involve examination of implementation and operational experiences and analysis of the impact of Tribal management on process and outcomes.  As with the illustrative comprehensive evaluation of DHHS programs that may be operated by Tribes under a potential Self-Governance demonstration, Tribes would identify unique health program objectives and quantitative evaluation of processes/outcomes would examine these unique objectives for each Tribe.

The comprehensive evaluation of IHS health programs managed under compacts could be conducted as a separate evaluation, or could be conducted as part of a comprehensive evaluation of all DHHS programs managed by Tribes under Self-Governance.  In the latter case, a subset of Tribes that participate in the new demonstration program and that currently compact for health could be used to evaluate health programs managed under compacts.

Tribes have been managing their health systems under compacts for a decade, so an evaluation of implementation of these programs and operational experiences would necessarily be retrospective in nature.  The retrospective nature of the evaluation might introduce some biases in the findings, but could provide useful information and insights for DHHS and Tribes.  Because nearly all Tribes and Tribal organizations that compact for health services report data to the IHS Resource and Patient Management System and these data are available for the pre-compact and post-compact period, little primary data collection would be necessary.

With respect to the feasibility issues of interest:

  • Tribal Support for the Evaluation Approach.  Tribes that manage health programs under compacts may not be supportive of a comprehensive evaluation of these programs, because they have been managing them for a number of years and they view health program Self-Governance as established and successful. Even if the evaluation was limited only to implementation and operational experiences and an evaluation of the extent to which Tribes achieved specific Tribally-set objectives, it is uncertain whether most Tribes would support or consider such an evaluation useful.
  • Availability of Appropriate Comparisons.  Assuming that the IHS RPMS data are available for participating Tribes and for the past 10 years, pre-post comparisons would be possible.  External comparisons could be based on a selected sample of IHS direct service Tribes or on aggregate IHS data that includes both compact and direct service Tribes.  However, either of those alternative external comparison groups is likely to raise concerns about appropriateness of the comparisons and value of the potential results.
  • Data Availability.  The availability of the RPMS and other IHS data would make it feasible to conduct the comprehensive evaluation of DHHS health programs and little primary data collection would be required.  Some people with whom the project team discussed the study expressed concerns, however, about the quality and completeness of the RPMS data.  It is possible that substantial work would be required to create the evaluation data base to ensure that the quality of the data was sufficient to produce reliable evaluation results.
  • Costs.[20] Assuming that the only primary data collection was to obtain information on satisfaction and experiences of patients using Tribally-managed and direct service facilities and site visits to 25 Tribes, and that the RPMS and other IHS data were available and usable, the evaluation of DHHS health programs managed under compacts could be conducted by an external evaluation contractor at a cost of approximately $750,000 to $1,000,000.
  • Trade-offs Between Comprehensiveness and Costs.  The costs of a comprehensive evaluation of DHHS health programs could be less if a decision was made not to collect primary data on patient satisfaction and experiences.  It also could be reduced if a smaller sample was examined for the evaluation (e.g. 15 Tribes rather than 25).  If both of these changes were made, the evaluation costs might be reduced to $500,000 to $750,000.

In summary, an evaluation of DHHS health programs operated by Tribes under compacts is technically feasible and the cost of such an evaluation would be moderate.  Given that Self-Governance of health programs has been in place for over a decade, however, Tribes might consider that the value of the evaluation findings could be limited.

Illustrative Limited Evaluation Model

A limited evaluation of Self-Governance of health programs could be designed that would focus on a few targeted issues that were identified by the Tribes and IHS as potentially useful.  Under the limited model, a few health indicators might be selected by compact Tribes and RPMS data could be used to identify compact Tribes that have been particularly successful in improving outcomes and meeting goals for those indicators.  Then, site visits could be conducted to identified Tribes and in-depth examination of their programs and processes that have been developed to achieve the successful outcomes that they have demonstrated.  These “best practices” and innovative programs could then be detailed and made available to other Tribes that might want to implement them.

It is likely that more Tribes would support a limited evaluation of this type since it would produce results that could be useful to other Tribes and could have the potential to improve health outcomes through dissemination of information on other strategies.  This approach would not require a formal comparison strategy and only RPMS and site visit data would be required to conduct the evaluation.  Costs of this type of limited evaluation would be modest – around $250,000 to $500,000 if an external evaluator conducted the evaluation.

Illustrative Evaluation Model Using Existing Aggregate Reporting Data

Self-Governance Tribes submit periodic narrative and financial reports to the Indian Health Service on their programs.  These reports could be used to construct profiles and trends over time on activities, objectives, and financial status of compact Tribes.  These limited data could provide information for an aggregate evaluation that would describe the experience over time of compacting.  It also might be possible, using these reports and other information within IHS to examine differences between Tribes that have successfully operated health programs under compacts for a number of years and Tribes that may have initiated planning or Self-Governance who subsequently withdrew from Self-Governance of IHS health programs.

Tribal support for this approach might be less than for the limited evaluation, both because the potential value of the results would be less and because there could be concerns about using confidential financial data reported to the Indian Health Service for evaluation purposes. Data likely exists for this evaluation approach, but it is not clear whether it could be used for evaluation purposes nor it is certain that the information contained in reports to IHS would be consistent and comparable across compact Tribes.  If the data were available and contained potentially useful, consistent information, the costs of this evaluation approach might be in the $150,000 to $250,000 range.  

Table F.2:  Matrix of Feasibility Issues for Illustrative Evaluation Models for IHS Programs Operated by Tribes Under Self-Governance


Comprehensive Evaluation Model

Limited Evaluation Model

Evaluation Using Aggregate Monitoring and Reporting Data

Tribal Support for Evaluation




Availability of Comparison Groups

Possible but difficulties



Data Availability

RPMS and site visits

RPMS and site visits

Data are all currently being reported. No new data requirements





Trade-off between comprehensiveness and usefulness of results and costs

Would produce reliable findings on a range of issues, but at high cost

Useful findings for a limited set of issues at modest cost

Limited findings at low cost