Escaping the Label Bottleneck: Self-Supervision, Transformers, Foundation Models

The label bottleneck

Labels are scarce, but raw neural recordings are comparatively abundant: idle resting-state EEG, continuous intracortical streams between task blocks, huge unlabelled archives. Self-supervised pretraining turns that abundance into a representation. You invent a pretext task solvable from the data alone, pretrain a large encoder on it, then fine-tune on your small labelled set.

This is the most credible route out of the small-data regime, and it is close kin to transfer learning: both reuse structure learned elsewhere so the scarce labels only have to specify the last mile rather than the whole map.

Pretext objectives: masked and contrastive

Masked modeling hides part of the signal — spans of time, whole channels, or discretised tokens — and asks the model to predict the hidden part from the rest. To fill the gap it must learn temporal and spatial structure: rhythms, cross-channel relationships, dynamics.

\mathcal{L}_{\text{mask}} = \sum_{i \in \mathcal{M}} \big\| x_i - \hat x_i \big\|^2 \quad\text{(or cross-entropy over discretised tokens for } i \in \mathcal{M})

Reconstruct only the masked set M; the loss forces the encoder to model the signal's internal structure.

Hide part of the signal and train the model to fill in the blanks. Because it can only succeed by understanding how the signal hangs together, this teaches useful structure with no labels at all.

\mathcal{M}: The set of masked (hidden) positions.
x_i,\ \hat x_i: The true value and the model's reconstruction at position i.
\big\| x_i - \hat x_i \big\|^2: Squared reconstruction error, summed over the masked spots.

Like guessing a word blanked out of a sentence from the words around it — the idea behind masked modelling.

Contrastive learning takes the complementary approach: pull together the representations of two augmented views of the same segment, and push apart views of different segments. The learned representation becomes invariant to the augmentations and sensitive to what distinguishes segments.

\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp\!\big(\mathrm{sim}(q, k^{+})/\tau\big)}{\sum_{k} \exp\!\big(\mathrm{sim}(q, k)/\tau\big)}

The InfoNCE contrastive loss; the temperature tau sets how sharply positives are separated from negatives.

Pull a query close to its one matching positive and push it away from all the negatives. The model learns representations where related samples cluster together, and the temperature \tau controls how harshly negatives are pushed away.

q,\ k^{+}: The query representation and its true positive match.
\mathrm{sim}(q, k): Similarity (e.g. dot product) between the query and a key.
\tau: Temperature: a small \tau makes the contrast sharp.

Two augmented views of the same EEG snippet are positives; snippets from other trials are negatives — the basis of contrastive learning.

Transformers for neural data

Transformers replace recurrence with attention: every time step attends directly to every other, so long-range dependencies are one hop away instead of many recurrent steps, and computation parallelises across the sequence.

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V

Scaled dot-product attention: each query pools values weighted by learned similarity to keys.

Each position (a query) looks over all positions (the keys), scores how relevant each is, turns those scores into weights, and takes a weighted blend of their values. This lets the model pull information from anywhere in the sequence, not just nearby.

Q,\ K,\ V: Queries, keys, and values — all learned projections of the input.
Q K^\top: Every query's similarity to every key.
\sqrt{d_k}: A scaling factor that keeps the scores from getting too large.
\mathrm{softmax}: Turns the scores into attention weights that sum to 1.

In "the cat that chased the mouse," the word "chased" can attend strongly to "cat" — the engine of the transformer.

The Neural Data Transformer (NDT) applies exactly this to population spiking, inferring the latent state of all time steps in parallel — much faster at inference than the sequential rollout of LFADS, while capturing the same kind of latent dynamics through a masked-modeling objective over neurons and time.

Toward neural foundation models

The grand version of the bet is the neural foundation model: pretrain one large model across many subjects, sessions, and tasks, then adapt it cheaply to each new use. If it worked as well for brains as it does for text, a new participant might need minutes of calibration instead of hours.

Does it actually transfer?

Honestly: self-supervised pretraining helps most when labelled data is very scarce and the unlabelled pool is large and well matched to the target. As soon as a strong classical baseline already captures the structure, the gains shrink or vanish. The right way to see the effect is a label-efficiency curve — accuracy versus number of labelled trials — where pretraining shows up as a leftward shift.