Feasibility Study for the Evaluation of DHHS Programs That Are or May Be Operated Under Tribal Self-Governance. Findings: Evaluation Feasibility


The methodology for assessing the feasibility of evaluating DHHS programs that in the future may be operated by Tribes under Self-Governance included:  1) establishment of a Technical Working Group that provided guidance and input throughout the project; 2) extensive background review of current DHHS programs operated by Tribes under contracts and IHS and Bureau of Indian Affairs (BIA) contracts and compacts, including a review of the literature to identify and assess existing evaluations of Self-Governance programs and discussions with DHHS program staff to identify current reporting requirements; 3) site visits to five Tribes and one Tribal organization [2] to obtain information about data that are currently available to or produced regarding DHHS programs operated by the Tribes and the availability of staff who could provide background and history of the Tribal  management of DHHS programs; and 4) discussion groups with a larger number of Tribal representatives to obtain feedback and additional information.

Three alternative illustrative evaluation models were identified:  1) a Comprehensive evaluation model with qualitative and quantitative components and extensive data needs; 2) a Limited evaluation model, focused on a smaller set of key evaluation issues and with less extensive data needs; and 3) an Aggregate evaluation model, that would use routine program reports to examine processes, objectives, and outcomes of DHHS programs that may be managed by Tribes under Self-Governance.  These three models were then assessed for feasibility, based on the background information, site visit findings, discussion group findings, and input from the Technical Working Group.  Feasibility considerations included:

  • Availability of appropriate comparison groups or strategies.
  • Availability of existing data and/or the feasibility of collecting additional data needed for evaluation.
  • Total costs of the evaluation approach and the trade-offs between costs and the potential value of evaluation findings.
  • Tribal support for the evaluation approach, based on considerations that include potential costs to DHHS and participating Tribes and potential benefits that might be obtained.

Results of the feasibility assessment included:

  • All three of the evaluation models were determined to be technically feasible. That is, necessary data either are available or could be collected and appropriate comparison strategies could be constructed.
  • Costs would be quite high for the Comprehensive evaluation model for evaluation of DHHS programs under a potential new Self-Governance demonstration, but results would be rigorous and could provide a wide range of potentially useful information.
  • The Limited evaluation model is technically feasible and less costly than the Comprehensive evaluation model. It would likely produce useful results regarding the evaluation issues addressed.
  • The Aggregate evaluation model, using standard reporting data, is technically feasible and least costly. It would produce some useful findings, but less detailed information than the Comprehensive or Limited evaluation models.

The Technical Working Group reviewed the evaluation feasibility study findings and, as a group, agreed that each of the three evaluation models was feasible and might produce useful findings for the Tribes and for DHHS.  The estimated costs associated with the Comprehensive evaluation model were considered high, relative to the potential value of the findings that would be generated.  There was general agreement within the Technical Working Group that the Limited evaluation model or the Aggregate evaluation model would be preferred to the Comprehensive evaluation model, since either of these alternatives could produce useful results at a more modest cost.

The Technical Working Group also emphasized that many Tribes would be supportive of and willing to participate in an evaluation of DHHS programs that may be operated by Tribes under a new Self-Governance demonstration.  The design of such a demonstration should result from consultation between DHHS and Tribes on the specific issues to be addressed and data that would be needed.  TWG members also stated that, because it would be important for DHHS to provide financial support for the development of data systems needed for any evaluation, DHHS should consider those costs as part of the total costs of any evaluation that might be designed.   Consideration of whether such consultation and agreement would need to precede or could equally well follow the enactment of any possible demonstration authority by Congress is beyond the scope of this report.

Finally, an important feasibility issue is the timing of an agreement between DHHS and the participating Tribes on the evaluation issues that would be addressed and on the data that would be provided by participating Tribes for an evaluation.  This agreement would be essential prior to the implementation of a new demonstration program to assist participating Tribes to be aware of data requirements and to establish procedures to collect data from the initiation of a demonstration. In the absence of such an agreement prior to implementation of a demonstration program, it would be difficult to ensure that appropriate and necessary data would be available for an evaluation.