JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Tokenizing the Brain: Self-Supervised Objectives

How raw spikes and field potentials become tokens, and the three families of label-free objectives — masked, contrastive, generative — that learn a representation from them.

From features to tokens

In earlier volumes you either hand-built features (band power, covariance geometry) or let a compact CNN like EEGNet learn them end-to-end for one task. A foundation model instead needs a general tokenization: a way to cut continuous, multichannel activity into a sequence of discrete units a transformer can attend over, without committing to any task.

Three tokenization strategies dominate. Per-spike tokens (POYO) treat each detected spike as a token carrying a learned unit-identity and a timestamp — naturally sparse and channel-count-agnostic. Patch tokens cut LFP/EEG into short time-channel patches, as vision transformers patch an image. Discrete codes run activity through a learned vector-quantizer (LaBraM), giving a finite codebook — a literal alphabet of neural symbols. Which you choose shapes everything downstream; see spike tokenization and population-level embedding.

Masked modelling

The workhorse objective is masked modelling: hide part of the tokenized activity and train the network to predict it from the rest, exactly as BERT masks words or a masked autoencoder masks image patches. NDT masks spike bins across time; LaBraM masks its quantized codes; the model must learn temporal and cross-channel structure to fill the gaps. This is masked modelling of neural activity.

A compact task-supervised network like EEGNet is the pre-foundation-model baseline: it learns features for one task. A foundation model instead pretrains the representation label-free, then reuses it — EEGNet-style heads can sit on top.

\mathcal{L}_{\mathrm{mask}}(\theta)=\mathbb{E}\!\left[\sum_{t\in M}\big(\lambda_{\theta,t}-x_{t}\log\lambda_{\theta,t}\big)\right],\qquad \lambda_{\theta,t}=f_{\theta}\big(x_{\setminus M}\big)_{t}

Masked Poisson reconstruction (as in NDT): for masked bins M, predict a firing rate lambda from the visible context and score it with the Poisson negative log-likelihood of the held-out spike counts.

The Poisson term is not decorative: spike counts are non-negative integers with mean-linked variance, so a Poisson (or negative-binomial) likelihood is the principled reconstruction loss — a squared error would mismodel the noise you studied in Volume II.

Contrastive learning

The second family is contrastive representation learning. Rather than reconstruct, it pulls together embeddings of samples that should be similar (nearby in time, or sharing a behavioural label) and pushes apart the rest. CEBRA made this concrete for neuroscience, yielding embeddings that are consistent across animals and sessions — a route to a shared latent geometry.

\mathcal{L}_{\mathrm{InfoNCE}}=-\,\mathbb{E}\!\left[\log\frac{\exp\!\big(\mathrm{sim}(z,z^{+})/\tau\big)}{\sum_{j}\exp\!\big(\mathrm{sim}(z,z_{j})/\tau\big)}\right]

The InfoNCE objective: a positive pair (z, z+) is scored against a bank of negatives z_j, with temperature tau. The quality of the representation hinges on how you define a positive.

Generative objectives, and what the representation buys

A third family is autoregressive/generative — predict the next token of activity (Neuroformer), which additionally lets the model simulate plausible neural futures. Whatever the objective, the payoff is the same: a latent representation you can probe, linearly read out, fine-tune, or align across sessions. The rest of this track is about making that representation shared and adaptable.