stochastic optimal control and the value function

Stochastic optimal control is the theory of steering a random system to do as well as possible on average. You have a state X_t that you can influence through a control u_t — a decision you choose, in real time, based on what you have observed — and the dynamics are a controlled SDE, dX_t = b(X_t, u_t) dt + sigma(X_t, u_t) dB_t. You pay a running cost f(X_t, u_t) along the way and a terminal cost g(X_T) at the end, and you wish to minimise the expected total cost J(t,x; u) = E[ integral_t^T f(X_s, u_s) ds + g(X_T) given X_t = x ] by choosing the control strategy. This is the engine behind everything from autopilots and portfolio choice to inventory management — any setting where you act repeatedly under noise to optimise a long-run objective.

The central object is the value function V(t,x) = inf_u J(t,x; u), the best achievable expected cost from state x at time t if you act optimally from now on. It is a function of state and time, not of any particular control: it has already optimised out the decision. Two structural facts make it usable. First, an admissible control must be adapted — it can depend only on the past, not the future — and we typically seek a feedback (Markov) control u_t = a(t, X_t), a rule that reads the current state and returns an action. Second, V satisfies the dynamic programming principle, which says the value over [t,T] can be computed by optimising over a small first step and then using V at the end of that step; differentiating that principle yields the Hamilton-Jacobi-Bellman PDE for V.

The value function is the linchpin connecting the two great routes to a solution: the dynamic-programming route (HJB equation, a nonlinear second-order PDE, plus a verification theorem confirming a candidate is optimal) and the Pontryagin route (the stochastic maximum principle and backward SDEs, which characterise the optimal control through adjoint/costate processes). An honest caveat: classically V is assumed smooth enough to plug into HJB, but in general V is only continuous and the HJB equation must be read in the viscosity-solution sense; smoothness (and hence a clean verification theorem) holds under nondegeneracy (uniform ellipticity of sigma sigma^T) and regularity of the coefficients, not always. Also the minimisation defining V must actually be attained by some admissible control for an optimal policy to exist — existence is a theorem with hypotheses, not automatic.