The Illustrative Comprehensive Evaluation Model for DHHS programs that may be operated by Tribes under Self-Governance, if a demonstration were to be authorized by Congress, is designed to be comparable to comprehensive evaluations that have been conducted of other DHHS programs and new delivery strategies. It would examine the implementation of the demonstration program, operational characteristics and experiences of the program over several years, and would collect data to permit quantitative measurement of processes and outcomes associated with the demonstration. Both pre-post and external comparison groups would be structured to permit assessment of the impacts of the demonstration, relative to what would have occurred in the absence of the demonstration. Data necessary for the comprehensive evaluation would be extensive and primary data collection would be necessary to address some of issues of interest.
In addition to the illustrative model assumptions in Appendix D, an additional assumption was made that is likely to affect the feasibility of this evaluation model: for each DHHS program managed by a specific Tribe, the Tribe would set two priority goals/objectives. The evaluation of outcomes would examine whether the Tribe was able to achieve its self-determined goals/objectives, rather than Federally-determined standard sets of objectives being measured across all participating Tribes. This assumption reflects the uncertainty about whether a set of common goals and objectives would be developed by Tribes that would permit a standard evaluation approach to be implemented. The Technical Working Group discussed this issues and indicated that it might be possible to identify a limited set of common goals/objectives, but that there was uncertainty on this issue. The potential complexity of the DHHS-Tribal consultations that would be needed may affect the timeliness of resolving or implementing a demonstration project.
Feasibility considerations with respect to the Illustrative Comprehensive Evaluation Model for a new DHHS demonstration include:
- Availability of Appropriate Comparisons. Pre-post comparisons would be possible, for those Tribes that managed the relevant DHHS programs under contract prior to the demonstration. It would be necessary to construct a reliable pre-demonstration baseline for Tribes that did not manage the relevant programs prior to the demonstration. Primary data collection would likely be necessary to obtain baseline (pre-) information on eligible persons and services needed and obtained prior to the demonstration, for each relevant program. Appropriate and reliable external comparisons would need to be carefully designed to address issues such as greater funding available to State-managed programs, but methodologies could be designed to adjust the comparison group data for differences in funding and other factors.
- Data availability. The comprehensive evaluation would require extensive data collection during the demonstration period that is considerably beyond the current data collection and reporting submitted by Tribes that operate the relevant DHHS programs under contracts.
- Costs. The costs associated with a comprehensive evaluation of the potential DHHS programs that may be operated under a new Self-Governance demonstration, including primary data collection to establish baselines, primary data collection through at least three years of the demonstration, and analysis and reporting would likely range from $3 million to $4 million and could possibly be higher.
- Trade-offs Between Costs and Comprehensiveness. It would be possible to reduce the costs of a comprehensive evaluation by limiting the evaluation to include only Tribes that were managing the DHHS programs under contracts prior to the demonstration. This would avoid the necessity of primary data collection to establish a baseline for the evaluation for those Tribes that were not previously managing the new programs. Similarly, the costs could be less if a decision was made to select a subset of Tribes participating in the demonstration –e.g. only evaluate the DHHS programs for a sample of 15 Tribes, rather than the assumed 25. However, reducing the number of Tribes that are included in the evaluation would reduce the reliability and usefulness of the findings. These two changes might reduce the cost of the evaluation to between $2 million and $3 million.
- Tribal Support for the Evaluation Approach. It is likely that some – or most – Tribes would be willing to support and participate in the comprehensive evaluation, if the outcome measures were uniquely set by each Tribe and if costs of participation were low. On the other hand, if the Tribes were required to bear a significant cost for data collection and reporting and/or if Tribes were to be evaluated based on a standard set of Federally-determined outcomes, there would likely be less support for this approach. The potentially high total costs of a comprehensive evaluation, discussed below, also could reduce Tribal support for this approach.
In summary, it would be technically feasible to conduct a comprehensive evaluation of DHHS programs operated under a Self-Governance demonstration. However, the associated costs of a comprehensive evaluation would be high – even if it were conducted only for a representative sample of participating demonstration Tribes.