Feasibility Study for the Evaluation of DHHS Programs Operated under Tribal Self-Governance. 2.3 - Feasibility Considerations

11/30/2003

The feasibility of conducting an evaluation of Tribal self-governance – either of a new demonstration program for DHHS non-health programs, that may be authorized by Congress, or of self-governance of DHHS health programs -- would be dependent on a number of issues and considerations.  These include:

  • Tribal support and agreement to participate in the evaluation.  The likelihood that Tribes will support the evaluation and agree to participate will be affected by the goals of the evaluation, the nature and extent of consultation between the Tribes and DHHS on the goals and processes of the evaluation, and the costs and burden of participation to the Tribes.
  • Self-selection Issues.  Self-selection bias is an issue for any demonstration program, since the Tribes that volunteer to participate in the demonstration are likely to be different in some characteristics than those that do not participate.  However, because Tribes would also have the option of participating or not participating in an evaluation, the magnitude of the self-selection issue may be greater than in most evaluation. Evaluation feasibility would be affected by the expected number of Tribes that would volunteer to participate.
  • Appropriate Comparison Groups.  Agreement by DHHS and Tribes on appropriate and acceptable – to DHHS and to Tribes – comparison groups are critical to the feasibility of any evaluation of Tribal self-governance.  Evaluation research involves comparing a new program’s operations and impacts relative to what would have been observed in the absence of the new program.  Pre-post comparisons are generally accepted approaches, but do not take into account underlying trends and changes that may affect what is observed in the new program.  External comparison groups are usually defined and examined to adjust pre-post data for any outside trends that may affect programs. 
  • Data Availability.  The availability of data for the pre-post self-governance period is a necessary condition for conducting an evaluation of DHHS programs operated by Tribes under self-governance.  Similarly, if external comparison groups are to be used, comparable and consistent data must be available for the relevant time periods of the evaluation.
  • Costs to DHHS and to Participating Tribes.  Costs of any evaluation approach considered are an important consideration in assessing evaluation feasibility.  Some evaluation alternatives may involve much higher costs than others and, thus, might be prohibitive.  Trade-offs may be considered between the comprehensiveness and rigor of evaluation alternatives and costs and the potential value of the findings that may be produced.

In addition to these issues and considerations, the Technical Working Group and others who contributed to and provided guidance for this study emphasized that an evaluation of DHHS programs operated by Tribes under a potential demonstration and an evaluation of self-governance of health programs requires significantly different approaches.  With this in mind, the project team considered the feasibility of evaluating DHHS health programs under self-governance as a separate task within the study.

View full report

Preview
Download

"draftreport.pdf" (pdf, 169Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®