Fleiss’ Kappa

Authors

Affiliations

Nate Yomogida, B.S., SPT

Doctor of Physical Therapy

B.S. in Kinesiology

Chloë Kerstein, B.S., SPT

Doctor of Physical Therapy

B.A. in Neuroscience

Fleiss’ K is based on the concept that the observed agreement is corrected for the agreement expected by chance. Krippendorff’s alpha in contrast is based on the observed disagreement corrected for disagreement expected by chance¹.

From the available kappa and kappa-like coefficients we chose Fleiss’ K [10] for this study because of its high flexibility. It can be used for two or more categories and two or more raters. However, similarly to other kappa and kappa-like coefficients, it cannot handle missing data except by excluding all observations with missing values¹.

Use

For nominal data with no missing values both Fleiss’ Kappa and Krippendorff’s alpha can be recommended

Calculation

For the calculation of the expected agreement for Fleiss’ K,the sample size is taken as infinite, while for Krippendorff’s alpha the actual sample size is used¹

References

1.

Zapf A, Castell S, Morawietz L, Karch A. Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate? BMC medical research methodology. 2016;16:93. doi:10.1186/s12874-016-0200-9