Neural Data Transformer (NDT)
The Neural Data Transformer (NDT), introduced by Ye and Pandarinath (2021), is a transformer applied to populations of spiking neurons that infers smooth latent firing rates from binned spike counts in a single non-recurrent forward pass. It was proposed as a faster, parallelizable alternative to LFADS: where LFADS uses a recurrent variational autoencoder, NDT masks and reconstructs spike bins with self-attention, achieving comparable latent quality and behavioral-decoding accuracy at much lower inference latency — attractive for real-time closed-loop BCI.
Later variants (notably NDT2 and related spatiotemporal tokenizers) extended the idea across sessions, subjects and brain areas, treating each channel-time patch as a token, and became a template for neural foundation models on intracortical data. The open question they surface is whether attention's flexibility genuinely helps beyond what a well-tuned dynamical model already captures on stereotyped motor tasks.
On a monkey reaching task, NDT infers per-trial firing rates from binned spikes in a single forward pass and matches LFADS's decoding accuracy while running fast enough to sit inside a closed-loop control loop.
Replacing a recurrent VAE with masked self-attention trades sequential inference for parallel, low-latency inference.