Binomial Chance Level

The naive assumption that chance accuracy equals 1/(number of classes) is only the expected value; with finite test sets the accuracy of a truly random decoder is a random variable, binomially distributed, with meaningful spread. For N test trials and K classes the standard deviation of chance accuracy is sqrt(p(1 − p)/N) with p = 1/K, so small test sets have wide chance intervals — for a two-class problem with 40 test trials, accuracies up to about 65% can occur by chance at the 5% level.

The practical consequences: report the upper confidence bound of chance (from the binomial or a permutation null), not the point value 1/K; be skeptical of high accuracies on tiny test sets; and prefer information-based or agreement-corrected metrics that account for K and N. The binomial bound assumes independent test trials, so temporal dependence widens the true interval further.