the optional quadratic variation (square bracket)
The optional quadratic variation [M] is the pathwise, model-free measure of how much a process actually wiggled: the limit of sums of squared increments along ever-finer partitions. Where the angle bracket <M> is the predictable compensator built from Doob-Meyer (and needs integrability), the square bracket [M] is a genuine pathwise object that exists for ANY semimartingale, is the quantity that appears in Ito's formula, and reduces to a clean sum-of-squared-jumps plus continuous part. It is the more fundamental of the two brackets in the sense that it always exists; the angle bracket is its predictable projection.
Definition: for a semimartingale M, fix a partition 0 = t_0 < t_1 < ... < t_n = t of [0,t]. The sums sum (M_{t_{i+1}} - M_{t_i})^2 converge in probability, as the mesh tends to 0, to a limit denoted [M]_t — the optional (or square-bracket) quadratic variation. It decomposes as [M]_t = [M]^c_t + sum_{s <= t} (Delta M_s)^2, the continuous quadratic variation plus the sum of squared jumps; for a continuous martingale the jump sum is empty and [M] = [M]^c. There is also an integration-by-parts identity defining it: M_t^2 = M_0^2 + 2 integral M_{s-} dM_s + [M]_t, so [M] is exactly the correction term that makes the Ito product rule work. For square-integrable martingales, M_t^2 - [M]_t is a martingale, and [M] and <M> share the same expectation, E([M]_t) = E(<M>_t) = E[M_t^2] - E[M_0^2].
Why it matters: [M] is the engine of Ito's formula — the (dM)^2 term in d f(M) = f'(M) dM + (1/2) f''(M) d[M] is precisely d[M], which is why ordinary calculus is wrong for stochastic integrals and why for Brownian motion [B]_t = t produces the extra one-half-second-derivative term. The Burkholder-Davis-Gundy inequalities bound martingale moments by moments of [M]^(1/2). The crucial honest point is that quadratic variation is NOT the total variation: Brownian paths have infinite total variation (you cannot integrate against them pathwise in the Riemann-Stieltjes sense) yet finite quadratic variation [B]_t = t — this finiteness of the second-order variation, despite infinite first-order variation, is the entire reason stochastic calculus exists as a separate subject. And [M] equals <M> only when M is continuous.
For Brownian motion B, partition [0,t] into n equal pieces; the sum of squared increments sum (B_{(i+1)t/n} - B_{it/n})^2 has mean t and variance of order 1/n, so it converges in L^2 to the constant t. Hence [B]_t = t — the quadratic variation is deterministic even though each path is nowhere differentiable.
[B]_t = t: finite quadratic variation despite infinite total variation — the fact that makes Ito calculus necessary.
Quadratic variation is not total variation: Brownian paths have infinite first-order (total) variation but finite second-order (quadratic) variation t, and it is this contrast that forces the extra second-order term in Ito's formula.