Multimodal (neural + behavior + language) model
A model that jointly represents neural activity together with other streams — behavior (kinematics, video, pose), sensory stimuli, and language. Two flavors recur. In fusion, the model conditions on several modalities to sharpen its representation or decoding: Neuroformer, for instance, is a GPT-style model over spikes together with behavior and stimulus. In alignment, neural representations are mapped into the embedding space of a pretrained model — CLIP-style vision embeddings, or a language model's token space — so that decoding can be open-vocabulary or generative, as in semantic language reconstruction from fMRI and image reconstruction via diffusion priors.
Honesty matters here because this is where the most striking demos live. Non-invasive semantic language decoding recovers gist and paraphrase, not verbatim inner speech, typically needs many hours of per-subject training, and degrades or fails if the subject resists. Image reconstructions are semantic and often subject-specific. The alignment to pretrained model spaces is a powerful bridge, but the outputs are reconstructions conditioned on strong generative priors, not direct transcriptions of thought.
When a decoder is coupled to a large generative prior, be careful attributing the output to the brain: the prior can hallucinate coherent content from weak neural evidence, so evaluation must control for what the prior alone would produce.