The Limited Evaluation Model would focus only on evaluation of new DHHS programs that would be managed by Tribes under a demonstration, if such a demonstration were authorized by Congress. The Limited Evaluation Model would include qualitative evaluation of implementation and operational experiences of participating Tribes and would attempt to identify effective management strategies and “best practices’ that would be useful to DHHS and to Tribes that are managing or considering managing programs under self-governance. Two outcome measures for each program would be identified by each Tribe, based on its priorities, and data collection and reporting would be limited to the data necessary to assess the extent to which each Tribe achieved its objectives for the selected outcome measures. Additional data would be drawn from quarterly or annual Federal reporting requirements for the programs that each Tribe manages. Comparison strategies would rely on pre-post comparisons and on national program benchmark data.
Feasibility considerations for the Limited Evaluation Model include:
- Willingness of Tribes to Participate. It is likely that many or most Tribes would be willing to participate in the Limited Evaluation, particularly if data collection was limited to only a few variables for each program and client and if DHHS provided training, software, and technical assistance to the Tribes for data collection.
- Availability of Appropriate Comparisons. Pre-post comparisons would be possible, for those Tribes that managed the relevant DHHS programs under contract prior to the demonstration. However, it would be difficult to construct a reliable pre-demonstration baseline for Tribes that did not manage the relevant programs prior to the demonstration. Primary data collection would likely be necessary to obtain baseline (pre-) information on eligible persons and services needed and obtained prior to the demonstration, for each relevant program. Appropriate and reliable external comparisons would need to be carefully designed to address issues such as greater funding available to State-managed programs, but likely could be constructed using aggregate national or federal program data.
- Data Availability. Primary data collection would be necessary to establish baseline information for Tribes that did not previously manage the relevant DHHS programs under contracts. In addition, data collection on two performance indicators/outcomes selected by each Tribe, for each program managed, would be necessary and one round of site visits would be conducted to each of 15-25 Tribes to collect qualitative information on implementation and operational experiences and innovative programs and “best practices.”
- Costs. The range of estimated costs for the Limited Evaluation Model is from $1 million to $2 million, assuming 15-25 Tribes would be involved in the evaluation.
- Trade-offs Between Comprehensiveness and Costs. The potential costs of the Limited Evaluation Model could be reduced if the participating Tribes were limited to those that had managed the relevant DHHS programs under contracts prior to the new demonstration. Costs could also be reduced if the sample of Tribes to be studied was reduced to 10, rather than 15-25. If both of these changes were made, then the range of estimated costs for the Limited Evaluation Model might be reduced to $750,000 to $1,500,000.
In summary, the Limited Evaluation Model is technically feasible and would involve moderate costs to carry out.