Feasibility Study for the Evaluation of DHHS Programs Operated under Tribal Self-Governance. 6.4 - Comprehensive Evaluation Model – Non-Health DHHS Programs

11/30/2003

The Illustrative Comprehensive Evaluation Model for new DHHS programs operated by Tribes under self-governance, if such a demonstration were to be authorized by Congress, is designed to be comparable to comprehensive evaluations that have been conducted of other new DHHS programs and initiatives.  It would examine the implementation of the demonstration program, operational characteristics and experiences of the program over several years, and would collect data to permit quantitative measurement of processes and outcomes associated with the demonstration.  Both pre-post and external comparison groups would be structured to permit assessment of the impacts of the demonstration, relative to what would have occurred in the absence of the demonstration.  Data necessary for the comprehensive evaluation would be extensive and primary data collection would be necessary to address some of issues of interest. 

In addition to the illustrative model assumptions in Appendix D, an additional assumption was made that is likely to affect the feasibility of this evaluation model:  For each DHHS program managed by a specific Tribe, the Tribe would set two priority goals/objectives.  The evaluation of outcomes would examine whether the Tribe was able to achieve its self-determined goals/objectives, rather than Federally-determined standard sets of objectives being measured across all participating Tribes.

Feasibility considerations with respect to the Illustrative Comprehensive Evaluation Model for a new DHHS demonstration include:

  • Willingness of Tribes to Participate.  It is likely that some – or most – Tribes would be willing to participate in the comprehensive evaluation, if the outcome measures were uniquely set by each Tribe and if costs of participation were low. On the other hand, if the Tribes were required to bear a significant cost for data collection and reporting and/or if Tribes were to be evaluated based on a standard set of Federally-determined outcomes, there would likely be very few Tribes that would agree to participate.
  • Availability of Appropriate Comparisons.  Pre-post comparisons would be possible, for those Tribes that managed the relevant DHHS programs under contract prior to the demonstration.  However, it would be difficult to construct a reliable pre-demonstration baseline for Tribes that did not manage the relevant programs prior to the demonstration.  Primary data collection would likely be necessary to obtain baseline (pre-) information on eligible persons and services needed and obtained prior to the demonstration, for each relevant program. Appropriate and reliable external comparisons would need to be carefully designed to address issues such as greater funding available to State-managed programs, but likely could be constructed.
  • Data availability.  The comprehensive evaluation would require extensive data collection during the demonstration period that is considerably beyond the current data reporting required of Tribes that operate the relevant DHHS programs under contracts.
  • Costs.  The costs associated with a comprehensive evaluation of the potential new DHHS programs demonstration, including primary data collection to establish baselines, primary data collection through at least three years of the demonstration, and analysis and reporting would likely range from $3 million to $5 million and could possibly be higher.
  • Trade-offs Between Costs and Comprehensiveness.  It would be possible to reduce the costs of a comprehensive evaluation by limiting the evaluation to include only Tribes that were managing the new DHHS programs under contracts prior to the demonstration.  This would avoid the necessity of primary data collection to establish a baseline for the evaluation for those Tribes that were not previously managing the new programs.  Similarly, the costs could be less if a decision was made to select a subset of Tribes participating in the demonstration –e.g. only evaluate the new DHHS programs for a sample of 10 Tribes, rather than the assumed 25.  These two changes might reduce the cost of the evaluation to between $2 million and $4 million.

In summary, it would be technically feasible to conduct a comprehensive evaluation of DHHS programs operated under a self-governance demonstration.  However, the associated costs of a comprehensive evaluation would be very high – even if it were conducted only for a representative sample of participating demonstration Tribes.

View full report

Preview
Download

"draftreport.pdf" (pdf, 169Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®