Feasibility Study for the Evaluation of DHHS Programs That Are or May Be Operated Under Tribal Self-Governance. Feasibility Considerations

03/03/2004

The feasibility of conducting an evaluation would be dependent on a number of issues and considerations.  These include:

  • Appropriate Comparison Groups.  Agreement by DHHS and Tribes on appropriate and acceptable – to DHHS and to Tribes – comparison groups are critical to the feasibility of any potential evaluation.  Evaluation research involves comparing a new program’s operations and impacts relative to what would have been observed in the absence of the new program.  Pre-post comparisons are generally accepted approaches, but do not take into account underlying trends and changes that may affect what is observed in the new program.  External comparison groups are usually defined and examined to adjust pre-post data for any outside trends that may affect programs. 
  • Data Availability.  The availability of data for the pre-post demonstration period is a necessary condition for conducting an evaluation of DHHS programs operated by Tribes under a Self-Governance demonstration.  Similarly, if external comparison groups are to be used, comparable and consistent data must be available for the relevant time periods of the evaluation.
  • Costs to DHHS and to Participating Tribes.  Costs of any evaluation approach considered are an important consideration in assessing evaluation feasibility.  Some evaluation alternatives may involve much higher costs than others that might be prohibitive.  Trade-offs may be considered between the comprehensiveness and rigor of evaluation alternatives and costs and the potential value of the findings that may be produced.
  • Tribal Support for an Evaluation.  The likelihood that Tribes will support an evaluation will be affected by the goals of the evaluation, the nature and extent of consultation between the Tribes and DHHS on the goals and processes of the evaluation, and the costs and burden of participation to the Tribes. In addition, Tribal support may also depend on the total costs of an evaluation, even if these costs are fully borne by DHHS, relative to the potential value and benefits that may be obtained from the evaluation.

An additional important feasibility issue is the timing of an agreement between DHHS and the participating Tribes on the evaluation issues to be addressed and on the data that will be provided by participating Tribes for the evaluation.  This agreement would be essential prior to the implementation of the new demonstration program to assist participating Tribes to be aware of data requirements and to establish procedures to collect data from the initiation of the demonstration. In the absence of agreement prior to the demonstration and specification of necessary data, it would be difficult to ensure that appropriate and necessary data would be available for the evaluation