The Small Group Discussions were conducted to obtain input and information from a wider group of knowledgeable individuals on the extent to which the draft findings and preliminary conclusions, based on the site visits, “ring true” from the experiences and perspectives of discussion group participants. Discussions occurred at the NIHB and the Self-Governance conferences. A Summary of these findings and other comments and suggestions is provided in matrix form in Appendix C of this report. Generally, comments fell into the following eleven themes:
- Financial Data: There was considerable objection to the possibility of examining ‘total Tribal revenues’ as one measure for future evaluations of programs administered through Self-Governance. While participants could understand why an evaluation might examine expenditures for a specific program, most felt that total Tribal revenues were off-base and outside the scope of any evaluation. In particular, there were concerns that Tribal gaming revenues would be reviewed along with Tribal program revenues, opening a much larger issue.
- Political Concerns: There was concern that a political undercurrent among some Tribes could be exacerbated if future evaluations were to make comparisons between Self-Governance and non-Self-Governance Tribal programs. There was also fear that setting statistical evaluation standards would increase pressure on Tribes to produce higher numbers, and move Tribes toward regionalization, possibly against the wishes of individual Tribes. It was also noted that the purpose of Self-Governance is to allow Tribes the authority to develop programs and services based upon Tribal priorities and not Federal or across-program priorities. This would make across-the-board evaluations of multiple programs difficult.
- Employment: Some discussants said it was not clear what employment measures would reveal about a program’s success and why ‘turn-over’ is examined instead of employee interviews. Most discussants also said that Tribes do have salary data by position available in program budgets and were confused about the suggestion in the draft report that these data are not available.
- Data and Measures: There were concerns that “pre/post” measures must take into account the fact that Tribal programs do not receive the same level of support as States for many programs which may be compacted, such as TANF and Child Support Enforcement. In particular, States do not usually forward the State match for Federal programs to Tribes. It was also noted that the draft document appears to suggest that an “integrated MIS system” is lacking in Tribal programs, but fails to note that it is also missing in most State and Federal programs as well. There was concern that Tribes will be held to a higher standard than States for these programs. The report suggesting pre- and post comparisons to baseline measures should also recommend that the responsibility to collect the data generated by States or Federal systems must remain with the Federal government, and not place Tribes in the position of having to retroactively collect comparison data from States.
- Systematic Issues: It should be noted that many Tribes administer programs through Tribal organizations or consortia and data by individual Tribe may be difficult to identify. It should also be noted that many Tribes taking over Federal programs will lack the infrastructure to be evaluated fairly against State or other programs. There is also no uniform data reporting system across Tribes, and the report should not assume that such a system exists for making program comparisons across Tribes.
- Success Stories: Numerous comments were made about the opportunity to use evaluation process to document success stories, Tribal innovation, and creativity. A process to identify lessons learned and to share this information with other Tribes getting started would be helpful to overcoming challenges. A qualitative approach would also allow for Tribal communities to tell their stories and convey their own community and cultural values about each program. Values such as “local ownership”, “community participation” could be conveyed through this process.
- Evaluation Purpose/Better Focus: The purpose of evaluation should be better described in the final report. It is important that this report communicate that it studied the feasibility of evaluating DHHS programs operated via Self-Governance and not the evaluation of Self-Governance alone. This is an important distinction.
- Trust: It was suggested that the best way to reduce Tribal skepticism about future evaluations would be to convey trust in Tribal programs. Tribal Self-Governance already begins with an assumption of competence at the Tribal level. Tribes then can identify Tribal outcomes for future measurement. There is a fear about evaluation that if a Tribe performs poorly it will lose its Federal funding. This historic lack of trust should be understood and considered for future evaluation.
- Cost Implications: Several of the standards suggested in the draft document, such as across-the-board comparisons or integrated MIS systems imply significant costs. If these standards will be among those used to evaluate DHHS programs administered through Self-Governance, then the Federal government must bear the responsibility to cover these costs. Tribes should also be provided access to data resources, just as States are provided, for many of these programs, so that Tribal infrastructure can expand with these program requirements.
- Tribal Base: It is important to understand the base point at which many Tribes are beginning the take over of DHHS programs under Self-Governance. Many will be starting from a point of inadequate funding. Many will be using Self-Governance as a way to bring more creativity and collaboration to build up an under-funded Federal program. Many of the Tribes may already be participating in the data reporting process of the DHHS program and may have data available.
- Limitations: The limits of conducting evaluation should be well understood. For example, suggestions in the report about across-the-board designs may not be realistic under Self-Governance once Tribes begin to reprioritize and implement innovative or creative changes. An across-the-board data pool may no longer exist. It is also important to understand the challenges of working with small populations and small data pools.
Based on the input and comments obtained through the discussion groups, the project team made several revisions to the approach to the feasibility study, including: 1) Tribal financial data that could be included in an evaluation design were re-specified as program-specific; 2) evaluation components that examined staffing levels and staff turnover issues were eliminated; 3) an integrated MIS system across all programs was not incorporated into the evaluation design approach; and 4) more emphasis was placed on the costs and burden of data collection on Tribes. It should be noted however, that the TWG advises that if adequate funding for data systems infrastructures were to be available, Tribes would actively support initiatives to advance development of integrated MIS within their program operations.
The comments and issues raised in the discussion groups were consistent with the guidance that was provided throughout the study by the Technical Working Group members. Similarly, Tribal staff who participated in the site visits raised many of the same issues. The agreement among the TWG, participants in the discussion groups, and site visit interviewees provided considerable support and confidence that the project team and DHHS were obtaining consistent information and guidance for the study.