Feasibility Study for the Evaluation of DHHS Programs That Are or May Be Operated Under Tribal Self-Governance. Limited Evaluation Model

03/03/2004

The Limited Evaluation Model would include qualitative evaluation of implementation and operational experiences of participating Tribes and would attempt to identify effective management strategies and “best practices’ that would be useful to DHHS and to Tribes that are managing or considering managing programs under Self-Governance.  Two outcome measures for each program would be identified by each Tribe, based on its priorities, and data collection and reporting would be limited to the data necessary to assess the extent to which each Tribe achieved its objectives for the selected outcome measures.[17] Site visits would be conducted to a representative subset of participating Tribes to obtain information on implementation, operations, and innovative programs and “best practices.” During the evaluation period, participating Tribes would also submit narrative annual reports on services provided, persons served, and a limited number of other aggregate program measures.[18] Comparison strategies would rely on pre-post comparisons.

Feasibility considerations for the Limited Evaluation Model include:

  • Availability of Appropriate Comparisons. Pre-post comparisons would be possible, for those Tribes that managed the relevant DHHS programs under contract prior to the demonstration.  It would be necessary to construct a reliable pre-demonstration baseline for Tribes that did not manage the relevant programs prior to the demonstration.  Primary data collection would likely be necessary to obtain baseline (pre-) information on eligible persons and services needed and obtained prior to the demonstration, for each relevant program.
  • Data Availability.  Primary data collection would be necessary to establish baseline information for Tribes that did not previously manage the relevant DHHS programs under contracts.  In addition, data collection on two performance indicators/outcomes selected by each Tribe, for each program managed, would be necessary and one round of site visits would be conducted to each of 15-25 Tribes to collect qualitative information on implementation and operational experiences and innovative programs and “best practices.”
  • Costs.  The range of estimated costs for the Limited Evaluation Model is from $1.5 million to $2 million, assuming 25 Tribes would be involved in the evaluation and a three-year evaluation timeframe.
  • Trade-offs Between Comprehensiveness and Costs.  The potential costs of the Limited Evaluation Model could be reduced if the participating Tribes were limited to those that had managed the relevant DHHS programs under contracts prior to the new demonstration.  Costs could also be reduced if the sample of Tribes to be studied was reduced to 15, rather than 25.  If both of these changes were made, then the range of estimated costs for the Limited Evaluation Model might be reduced to $1 million to $1.5 million.
  • Tribal Support for the Evaluation Approach.  It is likely that many or most Tribes would be willing to support and participate in the Limited Evaluation, particularly if data collection was limited to only a few variables for each program and client and if DHHS provided funding, training, software, and technical assistance to the Tribes for data collection.

In summary, the Limited Evaluation Model is technically feasible, would produce useful results, and would involve moderate costs to carry out.