orthogonal group O(n), SO(n)
Think of the rigid motions of R^n that fix the origin: rotations and reflections, anything that keeps lengths and angles exactly as they were. The matrices that do this form the orthogonal group O(n). It is the symmetry group of the round sphere, and of Euclidean geometry itself.
Algebraically, A is orthogonal iff A^T A = I, equivalently A^-1 = A^T, equivalently the columns of A form an orthonormal basis. From A^T A = I we get det(A)^2 = 1, so det A = +1 or -1. The orthogonal matrices with det = +1 form the special orthogonal group SO(n) — these are the pure rotations; det = -1 means a reflection is mixed in.
Geometrically O(n) = { length-preserving linear maps }, because preserving the inner product <Ax, Ay> = <x, y> is the same as A^T A = I. SO(n) is the connected component containing the identity: you can rotate continuously back to doing nothing, but you cannot continuously deform a mirror flip into the identity, which is why O(n) has two pieces.
A caveat on size: O(n) is NOT n^2-dimensional. The n(n+1)/2 equations of A^T A = I (a symmetric matrix condition) cut it down to dimension n(n-1)/2 — for n = 3 that is 3, matching the three independent axes you can rotate about. It is also COMPACT (closed and bounded, since every entry has absolute value <= 1), which is why averaging over it behaves so well.
Every planar rotation is an element of SO(2); its columns are orthonormal and its determinant is exactly 1.
Rotation count: SO(2) is a circle (one angle), SO(3) is 3-dimensional (yaw, pitch, roll). The famous oddity SO(3) is not simply connected — a 360-degree rotation is not deformable to nothing, but a 720-degree one is. That topological fact is the whole reason SU(2) and quaternions appear.