Feasibility Study for the Evaluation of DHHS Programs That Are or May Be Operated Under Tribal Self-Governance. Evaluation Model Using Aggregate Reporting Data

03/03/2004

The Evaluation Model Using Aggregate Reporting Data is more limited in scope than the Limited Evaluation Model described above. It would rely primarily on data assembled for aggregate periodic reports submitted by Tribes participating in the demonstration. Tribes would submit periodic reports that would be developed through a negotiated process between DHHS and the Tribes, prior to the demonstration implementation. It is also anticipated that additional information for evaluation purposes would be available from DHHS demonstration program officers who would have ongoing contact with each of the demonstration Tribes.[19] No primary data collection and no client-specific data would be required for this approach. The model was designed to manage costs, provide ongoing reporting of program services, and would require limited effort on the part of the Tribes participating in the evaluation. The comparison strategy would simply be examination of changes in program operations and achievements for each Tribe over the demonstration period.

Key issues for assessing the feasibility of this Evaluation Model based on aggregate reporting data include:

  • Availability of Appropriate Comparisons. Comparisons would be conducted of individual Tribes’ operations and reported aggregate outcomes over time, based on the aggregate report data, for each relevant program.
  • Data Availability.  Tribes would submit, on a periodic basis, all the aggregate program data necessary for the evaluation. Tribes that are currently operating these DHHS programs under contract arrangements would have systems and experience with the data reporting formats and would continue to submit these reports, for relevant programs, throughout the evaluation period. Tribes that are not currently managing relevant programs under contracts would be provided training and technical assistance in compiling necessary data and completing the reporting formats.
  • Costs.  Costs of this Evaluation Model using aggregate reporting data would be relatively low – around $750,000 to $850,000 for a three-year evaluation timeframe. Costs would be primarily for preparing data collection protocols that concatenate the selected items currently being collected, technical assistance to Tribes for data collection, data entry, and analysis and reporting.
  • Trade-offs Between Comprehensiveness and Costs.  While the estimated costs of this evaluation approach are relatively low, the evaluation results would provide general information but less detail on innovative management approaches and “best practices.”  Results would primarily be limited to reporting on current program services and clients served, and probably could provide some information on maintenance of effort and different Tribal priorities.
  • Tribal Support for the Evaluation Approach:  Based on the site visits and the willingness of site visit participants to provide data and reports for our review similar to that which would be used by this model, it seems likely that many or most Tribes would be willing to support and participate in this evaluation approach.  Willingness to participate would likely be increased, if the aggregate reporting by program that is necessary for the evaluation was clearly indicated as limited to the evaluation study and would be eliminated at the end of the evaluation.

This model is technically feasible and less costly than the other illustrative evaluation models discussed above.  It would provide some useful information, but would not produce results that would be as rigorous or valuable as the comprehensive evaluation or the limited evaluation models.  However, it would be least burdensome to Tribes in terms of data collection and reporting, particularly for participating Tribes that were previously managing the relevant programs under contracts.  This model also might fail to capture changing program objectives that result from tribal re-design since it would be gathering data only on measures that were reported on prior to the Self-Governance demonstration project.