Cohen's Kappa (κ)

An agreement metric that corrects observed accuracy for the agreement expected by chance: κ = (p_o − p_e)/(1 − p_e), where p_o is observed accuracy and p_e is the accuracy expected if predictions were random given the marginal class frequencies. κ = 1 is perfect, κ = 0 is chance-level, and negative is worse than chance. Because it discounts the free accuracy available from class priors, kappa is far more informative than raw accuracy under class imbalance and across differing numbers of classes.

In BCI, kappa is the preferred single-number score for multiclass and imbalanced problems (it was the primary metric of the BCI Competition datasets) because it is comparable across K and across imbalance levels in a way accuracy is not. Its limitations: kappa depends on the class prevalence in the test set, so values from datasets with different marginals are not strictly comparable, and it summarizes a full confusion matrix into one number, hiding per-class behavior.