Gaussian Processes and Their Covariance

What makes a process Gaussian

We start where Volume I left a single random vector and push it to an entire indexed family. A real-valued stochastic process (X_t) for t in an index set T is a Gaussian process if every finite linear combination of its values is a one-dimensional normal variable — equivalently, if for every finite set of indices t_1, ..., t_n the vector (X_{t_1}, ..., X_{t_n}) has a multivariate normal distribution. The index set T can be time [0, infinity), or an interval, or a region of R^d, or something far more abstract; nothing in the definition cares. What matters is that Gaussianity is imposed on every finite-dimensional snapshot at once.

The reason this definition is the right one — and not, say, "each X_t is individually normal" — is that demanding all linear combinations be normal is exactly what closes the class under the operations we care about. Sums of jointly Gaussian variables stay Gaussian, conditional expectations of one part given another are linear, and limits in L^2 of Gaussian variables are again Gaussian. A family in which each X_t is marginally normal but the pairs are not jointly normal can behave wildly; the joint requirement is what gives us a theory rather than a coincidence. From the stability of normals under addition you already have the smallest instance of this closure.

Mean and covariance: the whole law in two functions

Here is the central miracle of the Gaussian world. Attach to the process two functions: the mean function m(t) = E[X_t] and the covariance function (also called the kernel) k(s, t) = Cov(X_s, X_t) = E[(X_s - m(s))(X_t - m(t))]. Because every finite snapshot is multivariate normal, and a multivariate normal is determined completely by its mean vector and covariance matrix, the pair (m, k) determines the law of every finite-dimensional marginal — and hence, as the next section makes precise, the law of the whole process. This is the mean-and-covariance description of a Gaussian process, and it is why these objects are so beloved: an infinite-dimensional random object is captured by ordinary functions of one and two variables.

It pays to separate the two pieces by their roles. The mean m(t) is the deterministic backbone you can subtract off at will — centering, replacing X_t by X_t - m(t), leaves a zero-mean Gaussian process and loses no structural information, so almost all of the theory is developed for centered processes. The covariance k is where every interesting question lives: it encodes how strongly the value at s constrains the value at t, and through it flow smoothness of sample paths, the geometry of the reproducing kernel Hilbert space that the next guide builds, and the concentration estimates near the end of this rung. When people say "the Gaussian process," they almost always mean a fixed centered process and they are really pointing at its kernel.

Gaussian process X = (X_t), index set T
----------------------------------------
  Law fixed by:   m(t) = E[X_t]            (mean function)
                  k(s,t) = Cov(X_s, X_t)   (covariance / kernel)

Finite snapshot (t_1,...,t_n):
  (X_{t_1},...,X_{t_n}) ~ Normal( mu, Sigma )
      mu_i    = m(t_i)
      Sigma_ij = k(t_i, t_j)

Kernel MUST be symmetric positive semidefinite:
  k(s,t) = k(t,s),   and for all c_1,...,c_n in R:
  sum_{i,j} c_i c_j k(t_i,t_j) = Var( sum_i c_i X_{t_i} ) >= 0

The Gaussian dictionary: the pair (m, k) sets every finite-dimensional law, and the only constraint on k is symmetry plus positive semidefiniteness — which is forced because that double sum is literally a variance.

Positive-definiteness: the one rule the kernel must obey

Not every two-variable function can be a covariance, and the constraint is sharp and necessary. Pick any indices t_1, ..., t_n and any real coefficients c_1, ..., c_n and form the linear combination Y = sum_i c_i X_{t_i}. Its variance is non-negative by definition, and expanding gives Var(Y) = sum_{i,j} c_i c_j k(t_i, t_j) >= 0. So k must be a symmetric positive semidefinite kernel — every matrix [k(t_i, t_j)] you can extract is a symmetric positive semidefinite matrix. This is exactly the positive-definite condition, and it is the precise abstract content of "k is a legitimate covariance."

The converse direction is the constructive engine of the whole subject, and it is best stated honestly as a theorem rather than waved at. Given any symmetric positive semidefinite kernel k on T (and any mean function m), there EXISTS a Gaussian process with that mean and covariance — the law it must have on each finite snapshot is the prescribed multivariate normal, and these laws are mutually consistent precisely because the same kernel restricts compatibly to every sub-collection of indices. So positive-definiteness is not merely necessary; together with consistency it is sufficient. The work of actually producing the process from the consistent family is what the existence theorem in the next section handles.

From a kernel to a process: Kolmogorov's existence theorem

We have a recipe for the law of every finite snapshot; we need a single probability space carrying a genuine random function whose snapshots match. The bridge is the Kolmogorov existence (extension) theorem. It says: given a family of finite-dimensional distributions, one for each finite subset of the index set T, that is consistent — meaning the distribution on a larger finite set restricts (marginalizes) to the prescribed distribution on any smaller one, and is invariant under permuting coordinates — there exists a probability measure on the product space R^T whose finite-dimensional marginals are exactly the prescribed family. The coordinate maps on that space are then the process X_t.

Start with the data: an index set T, a mean function m on T, and a symmetric positive semidefinite kernel k on T x T.
For each finite tuple (t_1, ..., t_n) declare the snapshot law to be the multivariate normal with mean vector [m(t_i)] and covariance matrix [k(t_i, t_j)]; positive-definiteness is exactly what makes this matrix a valid covariance.
Check consistency: marginalizing a Gaussian over some coordinates again yields a Gaussian with the corresponding sub-mean and sub-kernel, so the family is automatically consistent — this step is free precisely because we built every law from one shared kernel.
Invoke Kolmogorov extension to get a single measure on R^T; the coordinate process under it is a Gaussian process with the desired mean m and covariance k. The construction is canonical and requires no continuity of paths.

Be honest about what this theorem does and does not give you. It manufactures the process as a measurable object on the product space, but the product sigma-algebra on R^T only controls events that depend on countably many coordinates — so quintessentially uncountable events like "the path is continuous" or "sup_t X_t <= a" are not even measurable in this raw construction. Getting a version with continuous (or otherwise regular) sample paths is a separate problem, solved by Kolmogorov's continuity criterion and, for Gaussian processes specifically, by entropy conditions on the kernel; that is the subject of Gaussian sample-path continuity criteria. Existence of the law and regularity of the paths are two genuinely different theorems.

A gallery of kernels you already know

The abstraction comes alive once you see that the processes from Volume I are just particular kernels. Brownian motion B_t on [0, infinity) is the centered Gaussian process with k(s, t) = min(s, t): increments are independent and Var(B_t) = t fall straight out of this single formula, which is why one can take Brownian motion as a Gaussian process as a working definition. The Brownian bridge on [0, 1] is the centered Gaussian process with k(s, t) = min(s, t) - s t, pinned to zero at both ends; it is exactly B conditioned to return to 0 at time 1, and you met it in the bridge, sheet, and time-inversion discussion. Conditioning a Gaussian process on some of its values is again Gaussian, and the bridge kernel is what that conditioning produces.

A second family is the stationary Gaussian processes, where the kernel depends only on the lag: k(s, t) = C(t - s). Here the Ornstein-Uhlenbeck process is the canonical example, with C(h) = (sigma^2 / 2 theta) e^(-theta |h|) — a smooth exponential decay of correlation that makes it the only stationary Gaussian Markov process on the line. Stationarity meshes beautifully with Fourier analysis: Bochner's theorem says a continuous stationary covariance C(h) is the Fourier transform of a finite measure, the spectral measure, so the process decomposes into uncorrelated random oscillations across frequencies. That is the spectral representation of stationary Gaussian processes, and it turns questions about correlation in time into questions about energy across frequency.

With the dictionary in hand, the rest of this rung reads as a tour of one object from several angles. The next guide canonizes the feature space hiding inside the kernel into the reproducing kernel Hilbert space — the natural home of the deterministic functions a Gaussian process can shadow. After that, the first chaos isometry identifies the random variables generated by the process with that very Hilbert space, leading to the isonormal process and Wiener chaos. Then Cameron-Martin describes exactly which deterministic shifts leave a Gaussian measure equivalent, and the final guide turns all this geometry into sharp concentration and supremum bounds. Every one of those chapters is, at bottom, a statement about the covariance kernel you have just learned to read.