What counts as evidence
Outside the lab, evidence standards are what stand between a genuine capability and a marketing claim. Consumer neurotech evidence standards borrow from clinical research: pre-registration, adequate sample size, and above all a proper control. Many neurofeedback and stimulation claims evaporate under a sham control with blinding, because expectation and placebo are powerful and a user who believes the device works will often behave as if it does. If a study has no blinded sham, treat its effect as unproven, not disproven — but unproven.
Above chance, honestly
Real-world datasets are small and imbalanced, which makes it easy to fool yourself. Two disciplines matter. First, compute a real chance level: for a small test set, the binomial chance level is above the naive 1/N, and a permutation test gives a distribution-free significance threshold. Second, guard against data leakage: with cross-validation on neural data you must respect the structure of the data — never let samples from the same trial, or the same subject, sit in both train and test — or you will report an accuracy the world will never reproduce.
A normal approximation to the above-chance threshold for n test trials and N classes: with few trials the critical accuracy sits well above 1/N. Reporting accuracy without this correction is a classic small-sample overclaim.
Metrics that mean something
Accuracy alone is a poor summary of a communication device. The information transfer rate (ITR) combines accuracy, the number of choices, and speed into bits per minute — the closest thing the field has to a shared currency, though it flatters synchronous paradigms and ignores usability. For control tasks, Fitts's-law throughput measures effective bits by how fast a user hits targets of varying difficulty, capturing the speed-accuracy tradeoff a static accuracy hides.
Wolpaw's bits per selection for N equiprobable targets at accuracy P; multiply by selections per minute for ITR. Note how the reward for adding choices (log N) is quickly erased if accuracy P falls — the honest tension between speed and reliability.
The metric no benchmark shows: adherence
The most decisive real-world metric never appears on a leaderboard: whether people keep using the device at all. Long-term adherence and human factors — setup time, comfort, cognitive effort, how gracefully it fails — often matter more than a few points of accuracy. A device that is a little less accurate but can be self-applied and worn all day will outperform, in aggregate lived benefit, a slightly better one that sits in a drawer.
This is why deployment, not just decoding, is a research problem. At-home, self-administered BCI and demonstrations of independent operation — including implanted users controlling digital devices without a technician present — are among the field's most meaningful real-world milestones precisely because they are measured in daily life, not in a shielded room.