Draft: This page has not been reviewed by a statistician yet. Use it as a starting point and check important choices with a statistician.

Cohen's kappa

Measures agreement between two raters or methods that assign categories to the same patients, beyond what chance alone would give.

Clinical example

Two radiologists independently classify the same chest X-rays as normal, abnormal or uncertain.

When to use it

Two raters or methods assign categories to the same patients.
You want agreement corrected for chance.
The categories are ordered (use weighted kappa) or unordered (use unweighted kappa).

Assumptions to check

Same items rated by both raters

Every patient or image is rated by both raters, independently of each other.

What to report

Kappa with 95% CI
The cross-table of both raters
The observed percentage of agreement

Common mistakes

Percent agreement isn't enough

Raters agree by chance some of the time. Report kappa, not only the percentage of agreement.

In your software

# R
library(irr)
kappa2(d[, c("rater_1", "rater_2")])
# Ordered categories: weighted kappa
# kappa2(d[, c("rater_1", "rater_2")], weight = "squared")

SPSS

CROSSTABS
  /TABLES=rater_1 BY rater_2
  /STATISTICS=KAPPA
  /CELLS=COUNT.

Sources

A confidence interval for kappa

The function kappa2() gives no confidence interval. A bootstrap confidence interval is one option: resample the patients (the pairs of ratings), recalculate kappa each time, and take the middle 95%.