Feasibility Study for the Evaluation of DHHS Programs That Are or May Be Operated Under Tribal Self-Governance. Design of Appropriate Comparison Groups

03/03/2004

Evaluation methodology requires that the impacts and outcomes of programs being evaluated be compared to the impacts and outcomes that would have occurred in the absence of the new program.  Design of appropriate comparison groups is a critical evaluation feasibility issue. 

Two types of comparison groups are generally used in a rigorous evaluation methodology:  1) pre-post comparisons to examine how the new program differs and what impacts it had, compared to the situation prior to the new program; and 2) external comparisons to control for underlying trends and changes that may affect the program being evaluated and the results produced by the evaluation. 

Many of the Tribes that might participate in a new demonstration program may already be managing the DHHS programs under contracts or grants.  For these Tribes, the pre-post comparison strategy would involve obtaining baseline data on the Tribes’ currently managed programs and then examining changes that occur as the Tribes’ operate the programs under Self-Governance.  If some participating Tribes did not manage specific programs under contracts or grants prior to the demonstration, there may be no “pre-” data for comparison at all or the “pre-” data may be only available for State-managed programs that may be more generously funded or otherwise inappropriate as a baseline for evaluating the program under Tribal management. It would be possible to collect baseline data on the extent to which Tribal members were served by the relevant DHHS programs prior to Tribal management.  The Technical Working Group members stated that this could be an important comparison, because services may not have been accessible or provided prior to the Tribe’s assuming responsibility for program management.

Appropriate external comparison groups may also be difficult to define for similar reasons, but could be constructed based on statistical methodologies developed to account for differences in funding levels and program objectives. Technical Working Group members and discussion group participants emphasized strongly that Tribal programs are often not funded at levels that are comparable to joint Federal-State funding for programs and that this must be accounted for in any design that includes external comparison groups.