Why model the sequence
Motor and speech intentions unfold over time, and their history matters. A per-window classifier treats each moment independently and throws that history away. A recurrent decoder instead carries a hidden state h_t that summarises everything seen so far, and updates it as each new sample of neural activity arrives.
h_t = \phi\!\left(W_h\,h_{t-1} + W_x\,x_t + b\right), \qquad \hat y_t = g\!\left(W_y\,h_t\right)The recurrence: gated variants (LSTM/GRU) replace phi with learned gates that keep gradients from vanishing over long sequences.
A recurrent network carries a running memory h_t that it updates each step from the previous memory and the new input, then reads out a prediction. Gated versions (LSTM/GRU) add learnable valves so this memory can persist over long stretches without the training signal fading.
- h_t
- The hidden state — the network's running memory at step t.
- x_t,\ \hat y_t
- The input and the output at step t.
- W_h,\ W_x,\ W_y
- Weight matrices that transform the memory, the input, and the read-out.
- \phi
- A squashing nonlinearity; learned gates replace it in LSTM/GRU.
Reading a sentence word by word, h_t is your evolving sense of the sentence so far — see the RNN decoder.
There is a deep symmetry here worth savouring: the very same recurrent equations are used both to model how motor cortex generates movement and to decode it. The dynamical-systems view of the brain and the sequence-model view of the decoder are two readings of one piece of mathematics.
The handwriting and brain-to-text BCIs
The clearest triumph of recurrent decoding is the handwriting BCI: a participant attempts to handwrite letters, an RNN reads the intracortical activity and emits a stream of character probabilities, and typing rates reach the neighbourhood of natural handwriting speed. Brain-to-text and speech decoders extend the same recipe to whole words.
But there is a labelling problem. The neural time series is long and sampled at, say, 100 Hz; the label — the text — is short and unsegmented. You do not know which time steps produced which character. Frame-by-frame supervision would require a hand alignment you do not have.
CTC: labelling without alignment
Connectionist temporal classification (CTC) dissolves the alignment problem. It adds a special blank symbol, defines a collapse map \mathcal{B} that removes blanks and repeats, and then defines the probability of a target as the sum over all frame-level paths that collapse to it.
p(y \mid X) = \sum_{\pi \in \mathcal{B}^{-1}(y)} \prod_{t=1}^{T} p(\pi_t \mid X), \qquad \mathcal{L}_{\text{CTC}} = -\log p(y \mid X)The sum over alignments is computed efficiently by a forward–backward dynamic program; you never need a hand alignment.
When you know the target sequence (say the letters) but not exactly which time step each one lines up with, CTC scores a transcript by summing the probability of every possible alignment. You never label the timing by hand.
- p(y \mid X)
- Probability of the target label sequence y given input X.
- \pi \in \mathcal{B}^{-1}(y)
- One frame-by-frame alignment; the set is all alignments that collapse to y.
- \prod_{t=1}^{T} p(\pi_t \mid X)
- Probability of one alignment, as the product over time steps.
- \mathcal{L}_{\text{CTC}}
- The training loss, the negative log-probability of the target.
The target "cat" could be emitted as "c-a-a-t", "cc-a-t-", and many more; CTC adds up all of them.
CTC pairs naturally with a language model at decode time: the acoustic-style neural posterior proposes characters, and a language model reshapes them into plausible words, sharply cutting error rates. The same architecture underlies much of the modern speech-BCI frontier covered in the speech track.
LFADS: inferring latent dynamics
LFADS (Latent Factor Analysis via Dynamical Systems) takes a generative view: population spiking is generated by a low-dimensional nonlinear dynamical system. It is a sequential variational autoencoder — an encoder RNN infers an initial condition g_0 (and any inferred inputs), a generator RNN rolls out a latent state, and a Poisson emission turns that state into firing rates.
g_t = \mathrm{RNN}(g_{t-1}, u_t), \quad \lambda_t = \exp(W g_t), \quad x_t \sim \mathrm{Poisson}(\lambda_t); \qquad \max_{\theta}\ \mathbb{E}_{q}[\log p(x\mid z)] - \mathrm{KL}\!\big(q(z\mid x)\,\|\,p(z)\big)A dynamical generative model trained by maximising the evidence lower bound (ELBO): reconstruct spikes while keeping the latent posterior near its prior.
LFADS assumes the messy spike counts you record are generated by a smooth low-dimensional dynamical system plus Poisson randomness. It learns that hidden system by maximising the ELBO — reconstruct the spikes well while keeping the inferred latents close to a sensible prior.
- g_t
- The RNN's latent generator state producing the dynamics.
- \lambda_t
- The firing rate that drives the Poisson spike model.
- x_t \sim \mathrm{Poisson}(\lambda_t)
- Spikes drawn from that time-varying rate.
- \mathbb{E}_{q}[\log p(x\mid z)] - \mathrm{KL}\big(q \,\|\, p\big)
- The ELBO: reconstruction quality minus a penalty for straying from the prior.
Noisy single-trial spikes get denoised into a clean latent trajectory shared across trials.
The payoff is single-trial denoising: LFADS projects noisy spike counts onto the neural manifold and recovers a clean latent trajectory, often better than any amount of smoothing. Compare it to GPFA, which assumes a linear-Gaussian latent — LFADS is the nonlinear, dynamical successor, and its inferred trajectories are what modern population-dynamics analyses (and some decoders) run on.
When recurrence helps — and when it does not
Recurrent nets shine when temporal context is long and labels are sparse: speech, handwriting, and anything with sequential grammar. For short motor-imagery windows or continuous cursor velocity, the gain over a well-tuned linear Kalman or Wiener decoder can be marginal — and RNNs are harder to keep numerically stable and low-latency in a closed loop.