Why synthesize instead of type
Text is accurate, but a voice is more than its transcript. Direct synthesis aims to restore the user's own voice with prosody, emphasis and the fast give-and-take of conversation — a real-time speech BCI that streams sound as intention unfolds, rather than committing finished sentences. For many users the return of an expressive, personal voice is the point, not merely the message.
The synthesis pipeline
Synthesis chains two learned maps and a vocoder. A neural network turns neural features r_{1:T} into an intermediate acoustic representation — a mel-spectrogram (or, via the articulatory bottleneck of Guide 2, vocal-tract kinematics first) — and a neural vocoder g_\phi reconstructs the waveform:
\hat{m}_{1:T} = f_\theta(r_{1:T}), \qquad \hat{s} = g_\phi(\hat{m}_{1:T}), \qquad \mathcal{L} = \sum_{t=1}^{T} \bigl\lVert m_t - \hat{m}_t \bigr\rVert_2^2Neural to mel to waveform. The decoder f_\theta is trained by regression to the mel target; the vocoder g_\phi (a WaveNet- or HiFi-GAN-style model) turns spectrogram frames back into audio.
Direct speech synthesis goes neural to mel-spectrogram to sound. A decoder learns to turn brain activity into spectrogram frames by matching a target, and a vocoder turns those frames back into an audible waveform.
- r_{1:T}
- The neural signal over time.
- \hat{m}_{1:T} = f_\theta(r_{1:T})
- The predicted mel-spectrogram from decoder f_\theta.
- \hat{s} = g_\phi(\hat{m}_{1:T})
- The waveform produced by vocoder g_\phi.
- \mathcal{L} = \sum_{t=1}^{T} \lVert m_t - \hat{m}_t \rVert_2^2
- Squared error against the true spectrogram.
The decoder is judged frame by frame on how close its spectrogram is to the real one — the goal of direct speech synthesis.
Closing the loop in real time
Everything changes when the loop is closed. Offline, you reconstruct audio from pre-recorded neural data and score it at leisure. Online, the user hears their own synthesized voice within a fraction of a second and adjusts — the very definition of a closed-loop BCI. That demands a strict latency budget across the whole chain:
t_{\text{loop}} = t_{\text{acq}} + t_{\text{feat}} + t_{\text{dec}} + t_{\text{synth}} \;\lesssim\; 50\text{--}100\ \text{ms}A rough real-time budget: acquisition, feature extraction, decoding and synthesis together should stay within a few tens of milliseconds per frame to feel immediate and support natural turn-taking.
For synthesised speech to feel immediate and allow natural back-and-forth, every stage — acquiring the signal, extracting features, decoding, and synthesising — has to add up to no more than a few tens of milliseconds per frame.
- t_{\text{loop}}
- Total round-trip latency per frame.
- t_{\text{acq}} + t_{\text{feat}} + t_{\text{dec}} + t_{\text{synth}}
- The four stage times: acquisition, features, decoding, synthesis.
If acquisition alone eats 40 ms, the other three stages must share what is left of the loop latency budget.
Tone, prosody and multilingual speech
Prosody is not decoration. In tonal languages — Mandarin, Cantonese, Taiwanese — pitch contour is lexical: the same syllable at a different tone is a different word. Tonal and multilingual decoding must therefore recover the laryngeal control that sets pitch, not just the segmental gestures. A text decoder can lean on the language model to disambiguate tone from context, but a synthesis decoder has to get the pitch physically right, or the output is unintelligible.
Silent and inner speech
Finally, the input side. A silent speech interface decodes attempted speech that produces no sound — the paralyzed-user case — and works well precisely because the motor drive to vSMC is still present and organized. Inner or covert speech, with no articulatory attempt at all, is a different and much harder animal: the signal is weaker, more idiosyncratic, and far less spatially structured, and current performance is correspondingly lower. The attempted-versus-imagined choice is the single most consequential design decision in a speech BCI — it sets both your ceiling and your ethics.