Brain-to-Text: Sequence Models and Language Priors

Speech as sequence transduction

Brain-to-text is a sequence-transduction problem: a stream of T neural time-frames must become a sequence of N symbols, with N \neq T and no frame-by-frame labels telling you which instant produced which character. Two model families dominate. Alignment-free models based on connectionist temporal classification marginalize over all possible alignments; attention-based sequence-to-sequence models learn the alignment as they go. The landmark imagined-handwriting BCI framed handwriting-to-text in exactly this way and reached typing rates approaching natural handwriting.

Connectionist temporal classification

CTC handles the missing alignment with a trick: add a blank symbol, let the network emit one label (or blank) per frame, then define a collapse function \mathcal{B} that removes blanks and merges repeats (so both `--h-e-l-llo-` map to `hello`). The probability of a target transcript \ell is the sum over every frame-level path that collapses to it:

p(\ell \mid X) \;=\; \sum_{\pi \,\in\, \mathcal{B}^{-1}(\ell)} \; \prod_{t=1}^{T} y^{\,t}_{\pi_t}

The CTC objective sums the probability of every alignment consistent with the transcript; a forward–backward recursion computes it efficiently, and its gradient trains the network end to end.

CTC scores a candidate transcript by adding up the probability of every way the network's per-frame outputs could line up to produce it. A forward-backward pass makes this sum cheap, and its gradient trains the whole network end to end — no hand-aligned timing needed.

p(\ell \mid X): Probability of transcript \ell given neural input X.
\pi \in \mathcal{B}^{-1}(\ell): One alignment; the set is all alignments that yield \ell.
y^{\,t}_{\pi_t}: The network's probability for symbol \pi_t at frame t.
\prod_{t=1}^{T}: Multiply across frames to score one alignment.

"hi" could be emitted as "h-h-i", "h-i-i", "_-h-i" and more; CTC sums them all.

CTC suits BCI well: it is monotonic (time only moves forward), needs no explicit segmentation of the neural stream into characters, and runs in a streaming, low-latency fashion — all properties a real-time speller needs.

Recurrent and transformer decoders

The network that emits those per-frame distributions has historically been an RNN decoder (a GRU or LSTM over binned firing rates or high-gamma features), and increasingly a transformer. Architecture matters less than the brutal constraint behind it.

A language model in the loop

The decoder proposes; the language model disposes. Formally, decoding is a search for the most probable text given the neural data, which factorizes into a neural term and a language prior:

\hat{w} \;=\; \arg\max_{w} \;\Bigl[\, \log p_{\text{dec}}(w \mid X) \;+\; \alpha\, \log p_{\text{lm}}(w) \;+\; \beta\, \lvert w \rvert \,\Bigr]

Beam search maximizes the neural log-probability plus a weighted language-model score, with an insertion bonus \beta to counter the blank-heavy bias of CTC. The weights \alpha, \beta are tuned on held-out data.

The best sentence isn't just what the neural decoder likes — it also has to sound like real language. Beam search picks the transcript that maximises the decoder's score plus a weighted language-model score, with a small bonus per inserted word to counter CTC's habit of emitting too many blanks.

\hat{w}: The chosen word sequence.
\log p_{\text{dec}}(w \mid X): The decoder's log-probability of the words.
\alpha\, \log p_{\text{lm}}(w): The language-model score, weighted by \alpha.
\beta\, \lvert w \rvert: An insertion bonus per word, weighted by \beta.

The decoder is unsure between "wreck a nice beach" and "recognise speech"; the language model tips it toward the latter.

# CTC beam search rescored by a language model (schematic)
beams = [('', 0.0)]                        # (text, score)
for t in range(T):
    probs = decoder_logits[t]             # distribution over chars + blank
    cand = []
    for text, score in beams:
        for c, p in top_k(probs, k):
            merged = collapse_ctc(text, c)
            s = score + log(p) + alpha * lm_logprob(merged) + beta * len(merged)
            cand.append((merged, s))
    beams = prune(cand, width)
best_text, _ = max(beams, key=lambda b: b[1])

A schematic CTC beam search rescored by a language model.

In practice the language model supplies a large share of the final accuracy, especially as vocabulary grows. But it cuts both ways: a strong prior can produce fluent, grammatical text that is not what the user tried to say — a genuine failure mode, and a red flag for authorship that Guide 5 returns to.

Metrics and vocabulary

How is performance measured? By word or character error rate, the edit distance between decoded and reference text normalized by reference length:

\text{WER} \;=\; \frac{S + D + I}{N}

Word error rate counts substitutions, deletions and insertions over the number of reference words; character error rate is the same over characters. Note WER can exceed 100% when insertions abound.

Word error rate adds up the words you had to substitute, delete, or insert to fix the transcript, divided by how many words the reference had. It's the standard scorecard; because insertions count, it can even exceed 100%.

S,\ D,\ I: Substitutions, deletions, and insertions needed.
N: The number of words in the reference transcript.
\frac{S + D + I}{N}: Total edits per reference word.

Reference "the cat sat" decoded as "the hat sat" is one substitution over three words, so \text{WER} \approx 0.33 — the word error rate.