Preserving the inner product
In Volume I an orthogonal matrix satisfied Q^T Q = I and had orthonormal columns. The real meaning: Q preserves the [[inner-product|inner product]], hence preserves lengths and angles. Indeed <Qu, Qv> = (Qu)^T (Qv) = u^T Q^T Q v = u^T v = <u, v>. A transformation that keeps the inner product is a rigid motion fixing the origin — exactly what we want a symmetry to be.
The orthogonal group O(n)
Collect all such matrices and you get the orthogonal group O(n) = { Q : Q^T Q = I }. It is a subgroup of GL(n) by our recipe: if Q^T Q = I and R^T R = I then (QR)^T(QR) = R^T Q^T Q R = R^T R = I, and Q^-1 = Q^T is orthogonal too. From Q^T Q = I we get det(Q)^2 = 1, so det(Q) = +1 or -1: O(n) splits into rotations and reflections.
# A 2D rotation by angle t and a reflection both live in O(2) R(t) = [cos t, -sin t; sin t, cos t] R^T R = I, det R = cos^2 t + sin^2 t = +1 (rotation, in SO(2)) F = [1, 0; 0, -1] F^T F = I, det F = -1 (reflection, NOT in SO(2)) # product of two reflections is a rotation: F1 = [1,0; 0,-1], F2 = [cos a, sin a; sin a, -cos a] F2 * F1 = R(a) det(+1)(-1)... = (-1)(-1) = +1
SO(n): the connected rotation group
Add the condition det = +1 and you keep only the rotations: the special orthogonal group SO(n) = O(n) intersect SL(n). SO(n) is the part of O(n) connected to the identity — you can rotate continuously from I to any rotation, but you can never reach a reflection without a jump. SO(2) is a circle of rotations; SO(3) is the group of all spatial rotations and is the symmetry group of a sphere.
Going complex: U(n)
Over the complex numbers the right inner product is <u, v> = u* v, where the star means conjugate-transpose. The matrices preserving it satisfy U* U = I and form the unitary group U(n). Unitary is to complex space what orthogonal is to real space: it preserves the complex norm and length. From U* U = I we get |det U| = 1, so det U is a unit complex number e^{i*theta} lying on the circle — richer than the mere +/-1 of the real case.