Feasibility Study for the Evaluation of DHHS Programs Operated under Tribal Self-Governance. 6.7 - Evaluation Model Using Aggregate Monitoring and Reporting Data

11/30/2003

The Evaluation Model Using Aggregate Monitoring and Reporting Data is more limited in scope than the Limited Evaluation Model described above. It would focus only on the new DHHS programs that would be included in a demonstration and relies primarily on data assembled for aggregate periodic reports submitted by Tribes participating in the demonstration. Tribes would submit periodic reports that would be developed through a negotiated process between DHHS and the Tribes, prior to the demonstration was implemented. No primary data collection would be required for this approach. The model was designed to manage costs, provide ongoing reporting of program services, and would require limited effort on the part of the Tribes participating in the evaluation. The comparison strategy would simply be examination of changes in program operations and achievements for each Tribe over the demonstration period.

Assessing its feasibility of this Evaluation Model based on aggregate reporting data, we find:

  • Willingness of Tribes to Participate:  Based on the site visits and the willingness of site visit participants to provide data and reports for our review similar to that which would be used by this model, it seems likely that many or most Tribes would be willing to participate.  Willingness to participate would likely be increased, if the aggregate reporting by program that is necessary for the evaluation was clearly indicated as limited to the evaluation study and would be eliminated at the end of the evaluation.
  • Availability of Appropriate Comparisons. Comparisons would be conducted of individual Tribes’ operations and aggregate outcomes over time, based on the aggregate report data, for each relevant program.
  • Data Availability.  Tribes would submit, on a periodic basis, all the aggregate program data necessary for the evaluation. Tribes that are currently operating these DHHS programs under contract arrangements would have systems and experience with the data reporting formats and would continue to submit these reports, for relevant programs, throughout the evaluation period. Tribes that are not currently managing relevant programs under contracts would be provided training and technical assistance in compiling necessary data and completing the reporting formats.
  • Costs.  Costs of this Evaluation Model using aggregate reporting data would be relatively low – around $500,000 to $750,000 for a three-year evaluation timeframe. Costs would be primarily for preparing data collection protocols that concatenate the selected items currently being collected, technical assistance to Tribes for data collection, data entry, and analysis and reporting.
  • Trade-offs Between Comprehensiveness and Costs.  While the estimated costs of this evaluation approach are relatively low, the results of the evaluation would be limited in detail and in usefulness.  Results would primarily be limited to reporting on current program services and clients served, and probably could provide some information on maintenance of effort and different Tribal priorities.

This model is technically feasible and less costly than the other illustrative evaluation models discussed above.  It would provide some useful information, but would not produce results that would be as rigorous or valuable as the comprehensive evaluation or the limited evaluation models.  However, it would be least burdensome to Tribes in terms of data collection and reporting, particularly for participating Tribes that were previously managing the relevant programs under contracts.

View full report

Preview
Download

"draftreport.pdf" (pdf, 169Kb)

Note: Documents in PDF format require the Adobe Acrobat Reader®. If you experience problems with PDF documents, please download the latest version of the Reader®