Reproducibility and overclaiming

BCI and neurofeedback research carry the same reproducibility burden as the rest of neuroscience, amplified by small samples, flexible analysis pipelines, publication bias toward positive results, and — on the consumer side — a commercial incentive to publicise. The recurring failure patterns are concrete: reporting a best subject or best session, testing many features or windows and presenting the one that worked, comparing against no realistic baseline, and leaking information from test into training so that offline accuracy overstates real performance.

The countermeasures are now well understood, if unevenly practised: open datasets and code, standardised benchmark suites that force new methods to be compared on common data, pre-registration and blinding, cross-subject and cross-session evaluation rather than within-session, and honest reporting of the field drop and of failures such as users who could not use the system. Overclaiming is the specific hazard of moving toward the public, where a laboratory effect of modest size can be restated as a product benefit; guarding against it is a scientific responsibility as much as a regulatory one.