the partition function
A single sum that quietly contains all the thermodynamics of a system. Add up e^(-E / k_B T) over every microstate and you get one number, Z. From its logarithm and derivatives you can then extract energy, entropy, pressure, heat capacity, everything. The German name Zustandssumme, 'sum over states,' says exactly what it is.
In the canonical ensemble Z = sum_i e^(-E_i / k_B T) = sum over states of e^(-beta E), with beta = 1/(k_B T), or an integral over phase space with the classical measure that carries a 1/(h^(3N) N!). It normalizes the Boltzmann distribution, p_i = e^(-beta E_i) / Z. The bridge to thermodynamics is F = -k_B T ln Z; then <E> = -d(ln Z)/d beta, S = -(dF/dT)_V, and P = -(dF/dV)_T. For independent subsystems Z factorizes: Z = z^N for distinguishable parts, or Z = z^N / N! for identical ones, where z is the single-particle partition function.
The slogan is that if you know Z, you know everything, because Z is the generating function of the ensemble. But it merely counts states, so it hides the physics until you differentiate it, and getting the measure right is essential: the 1/N! for identical particles and the h^(3N) cell size cannot be omitted, or you get the Gibbs paradox and a non-extensive entropy.
A two-level system with energies 0 and epsilon has Z = 1 + e^(-epsilon / k_B T); its average energy <E> = epsilon e^(-epsilon/k_B T) / (1 + e^(-epsilon/k_B T)) gives the Schottky heat-capacity peak.
From one small sum come the energy and the characteristic bump in heat capacity.
There is a partition function for each ensemble, canonical Z and grand canonical Xi, each linked to its own free energy (F = -k_B T ln Z, Phi = -k_B T ln Xi); do not confuse them.