What is chance, really?
For c balanced classes the naive chance level is 1/c — but that is the expected value, not a threshold. With few trials, a useless decoder scatters around 1/c with real spread, so an accuracy comfortably above 1/c can still be pure luck. The right question is: what accuracy would a coin-flip decoder exceed only \alpha of the time?
a_{\alpha}=\frac{1}{c}+z_{1-\alpha}\sqrt{\frac{\tfrac{1}{c}\big(1-\tfrac{1}{c}\big)}{N}}Normal approximation to the binomial chance level. For c=2, N=100 the 95% threshold is ≈ 0.58 — a decoder at 56% is not above chance, however tempting the number looks.
"Chance" is not exactly 1/c — with only finitely many trials a lucky classifier can score a bit above it. This gives the accuracy you must actually beat to be convincingly above chance.
- c
- Number of classes; naive chance is 1/c.
- N
- Number of test trials.
- z_{1-\alpha}
- The z-score for your confidence level (about 1.64 at 95%).
- a_{\alpha}
- The accuracy threshold that marks significance.
For a 2-class task with 100 trials the 95% threshold is about 0.58, so a decoder at 56% is not above chance.
Permutation testing: chance without assumptions
The binomial formula assumes independent trials and a clean chance level — both often false with cross-validation, imbalance, or temporal correlation. A permutation test sidesteps every assumption: shuffle the labels, rerun the entire pipeline, and record the score. Repeat many times to build the null distribution of scores your pipeline produces when there is genuinely nothing to decode.
\hat p=\frac{1+\#\{\,\pi:\ s_{\pi}\ge s_{\mathrm{obs}}\,\}}{P+1}The permutation p-value over P shuffles. The +1 counts the observed statistic itself, so p is never zero — an honest floor of 1/(P+1).
Shuffle the labels many times to see what accuracy pure luck produces; your real score's p-value is the fraction of shuffles that matched or beat it. The +1 keeps it honest so p is never exactly zero.
- P
- Number of label shuffles you run.
- s_{\pi}
- The score obtained on one shuffled dataset.
- s_{\mathrm{obs}}
- Your actual observed score on the real labels.
- \hat p
- The resulting permutation p-value.
If none of 999 shuffles beat you, the permutation p-value is 1/1000.
Beyond accuracy: kappa and the confusion matrix
Accuracy collapses a whole confusion matrix into one number and discards where the errors fall. Cohen's kappa corrects accuracy for the agreement expected by chance, giving a scale on which 0 means 'no better than guessing the priors' regardless of how many classes there are.
\kappa=\frac{p_o-p_e}{1-p_e}Cohen's kappa: observed agreement p_o discounted by chance agreement p_e. κ = 1 is perfect, 0 is chance, negative is worse than chance.
Accuracy alone flatters you when classes are imbalanced. Kappa subtracts the agreement you would get by luck and rescales, so 1 is perfect and 0 is no better than guessing.
- p_o
- Observed accuracy (the raw agreement rate).
- p_e
- Accuracy expected from chance alone.
- \kappa
- Chance-corrected agreement: 1 perfect, 0 chance, negative worse than chance.
90% observed but 80% expected by chance gives kappa of 0.5 — much less impressive than the raw number.
Class imbalance and the accuracy paradox
Many BCI paradigms are intrinsically imbalanced — a P300 speller sees far more non-targets than targets. Under class imbalance a decoder that always predicts the majority class can post a spectacular accuracy while being clinically useless: the accuracy paradox. Balanced accuracy (mean per-class recall), kappa, and the full confusion matrix expose it; raw accuracy hides it.
Imbalance also biases the classifier itself: LDA's \log\pi_k term and most loss functions favour the majority. Remedies — reweighting classes, adjusting the decision threshold, or matching the operating point to the application's cost of misses versus false alarms — are choices about utility, and they belong to the metric discussion, not buried in a default.