The write problem is an inverse problem
To make a brain see a scene, you must inject the activity the healthy visual pathway would have produced for that scene, then let the intact downstream cortex do the rest. Stated that way, artificial vision is the biomimetic-write problem: invert the visual system's encoding. Motor decoding is a well-posed forward map from neural data to intent; writing vision is its inverse, and inverses are hard — under-constrained, sensitive to model error, and entangled because one electrode drives many cell types at once.
The retinal encoder
A landmark idea reframed the problem. Instead of relaying raw pixel brightness to the electrodes, first pass the camera image through a model of the retina's own neural code — the ganglion-cell spike trains a healthy retina would have generated — and then stimulate to reproduce those spikes. In simulation, this encoding-model approach reconstructed images dramatically better than the raw-pixel encoding used by early devices. The lesson: the electrodes should speak the retina's language, not the camera's.
A linear-nonlinear-Poisson (LNP) encoding model of a ganglion cell: convolve the stimulus s(t) with a spatiotemporal filter k, pass through a static nonlinearity (here exponential) and a bias b to get an instantaneous firing rate lambda(t), then draw spikes as a Poisson process. The encoder's job is to make the stimulated retina fire this same lambda(t).
The caveat is honest and important. Reproducing the code assumes you can drive each cell type independently and that you know the ground-truth filters for each cell — but electrical stimulation recruits many cell types indiscriminately, so the encoder is far easier to state than to deliver. This is precisely why cell-type-specific writing (optogenetics, in the next guide) is so attractive: it is a route to actually delivering the code the encoder computes.
Scene simplification and gaze
With only tens to a few hundred usable phosphenes, you cannot render a full scene — so you must choose what to show. Scene simplification puts computer vision in front of the electrodes: edge detection, object saliency, depth thresholding, and highlighting of faces or text decide which handful of phosphenes will carry meaning. In effect the camera and its AI do the perceiving, and the electrodes convey a curated summary — a caption, not a photograph.
Because a head- or glasses-mounted camera does not move with the eye, where the user looks must actively drive what is written. Gaze-contingent rendering uses eye-tracking to re-map the image onto the cortical representation of wherever the fovea is currently pointed, so that percepts move naturally with gaze rather than sliding around independently — the antidote to the gaze-desynchronization problem raised at the end of the last guide.
Retinotopy and the warped map
The map from visual field to cortex, retinotopy with cortical magnification, is smooth but highly non-uniform, so a regular electrode array lands on an irregular patch of visual space. Establishing which electrode corresponds to which field location and phosphene shape is therefore a per-subject calibration — phosphene mapping — done psychophysically after implantation, electrode by electrode, by asking the person what and where they see.
Because that map is individual and drifts over time, the encoder cannot be shipped fixed; it must be fit to each user and re-checked. That connects visual prosthetics straight to the adaptive-decoder and representational-drift problems that run through the rest of Volume III — a visual prosthesis is a closed loop that must keep re-learning its own output.