Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. APPENDIX M: ACI Index

05/01/2019

ACI Index

An analysis of the marginal Kappa distributions suggests that the lower values of Kappa may be attributed to the so-called "Kappa paradoxes" (Feinstein and Cicchetti, 1990), which occur when raters yield a high percent positive (responses in which both raters give a rating of "yes") or negative agreement (both raters give a rating of "no"). Simply put, Kappa tends to yield a low value when the raters show high agreement, which is counterintuitive since one would expect a higher reliability when two qualified raters reach high agreement in observed ratings. Gwet's adjusted chance-corrected AC1 index (Gwet 2008) was developed specifically to overcome these weaknesses of the Kappa statistic.

Gwet's agreement co-efficient can be used in more contexts than Kappa because it does not depend upon the assumption of independence between raters. The AC1 is based upon the more realistic assumption that only a portion of the observed ratings will potentially lead to agreement by chance.[8] Gwet (2008) indicates that a reasonable value for chance-agreement probability should not exceed 0.5, whereas chance-agreement probability for Cohen's (1960) Kappa can be any value between 0 and 1. For instance, if the raters agree 90 percent of time, Gwet's AC1 would assume that chance-agreement should be at most 50 percent, whereas Kappa would calculate the chance-agreement at 81 percent on the positives and 1 percent on the negatives for a total of 82 percent.

This limit of chance-agreement prevents Gwet's AC1 statistics from the form of erratic behavior seen in Kappa. Another beneficial property of AC1 is that while Cohen's Kappa penalizes raters who produce similar ratings or marginal distributions, such penalization does not exist in AC1; on the contrary, raters with homogenous marginal distributions are rewarded by AC1.