The problem: infinitely many numbers, finitely many bits
By now you know that every numerical answer is an approximation, and one big reason starts before any algorithm runs: the moment you store a real number, you have already rounded it. Between 1 and 2 there are infinitely many real numbers, but a fixed-width chunk of memory can hold only finitely many distinct bit patterns. A 64-bit word, for example, has exactly 2^64 patterns — a huge number, but still finite. So the computer cannot keep 'the real line'; it keeps a carefully chosen finite grid of stand-ins, and snaps your number to the nearest grid point.
One obvious design would be a fixed-point representation: pick, say, 6 digits before a decimal point and 4 after, giving an evenly-spaced grid like 12.3456, 12.3457, and so on. That is fine for money but terrible for science, where you want to talk about both the radius of a proton (about 10^-15 m) and the distance to a galaxy (about 10^22 m) with the same variable. An evenly-spaced grid fine enough for the proton would need astronomically many digits to reach the galaxy. We need a grid whose spacing grows with the size of the number.
Scientific notation, in base 2
The fix is the one you already use by hand: scientific notation. You write 6.022 × 10^23 instead of a string of 24 digits — a significand (6.022) times a power of ten set by an exponent (23). Floating point is exactly this, but in base 2, because the hardware speaks bits. A real number x is stored as x ≈ (-1)^s × m × 2^e, where s is a single sign bit, m is the significand (also called the mantissa), and e is the exponent. The 'point' floats: increase e and the same significand lands on a coarser part of the grid, decrease e and it lands on a finer part.
There is a neat trick to squeeze out one extra bit. A normalized number in base 2 is written so the significand is 1.something — exactly one nonzero digit before the point, and in binary the only nonzero digit is 1. So that leading 1 is always there; we don't bother storing it. This 'hidden bit' means a significand field of 52 stored bits actually carries 53 bits of precision. The cost is that the spacing between representable numbers doubles every time the exponent steps up by one: numbers near 1000 are spaced about 1000 times farther apart than numbers near 1. That is the whole point — relative precision, roughly the same number of significant figures everywhere.
IEEE 754: one agreed format for everyone
If every chip vendor invented their own significand width and rounding rules, the same program would give different answers on different machines — a nightmare for floating-point arithmetic and for reproducibility. Since 1985 the IEEE 754 standard has fixed the bit layout, the rounding behaviour, and the special values, so that 1.0 + 1.0 means the same thing on your phone, your laptop, and a supercomputer. Two sizes dominate everyday computing, and the bit budget tells the whole story.
single (float32): 1 sign | 8 exp | 23 frac -> 24-bit precision, ~7 decimal digits double (float64): 1 sign | 11 exp | 52 frac -> 53-bit precision, ~16 decimal digits value = (-1)^s * (1.frac)_2 * 2^(e - bias) [normalized] bias = 127 (single) bias = 1023 (double)
Double precision (often just called a 'double') gives you about 15-16 significant decimal digits, and it is the default for almost all scientific work. Single precision gives only about 7 digits but halves the memory and is much faster on GPUs, which is why machine learning leans on it. Keep this number in your pocket: ~16 digits is your entire budget in a double. That budget is not free spending money — later guides show how a badly-conditioned problem can spend most of it before your algorithm even starts, so treat those digits as precious.
Reading a number off the bits
Let's decode a tiny example by hand so the format stops feeling magical. Take the number 6.5 in single precision. The recipe is just binary scientific notation, followed by packing the three fields.
- Write 6.5 in binary: 6 is 110 and 0.5 is 0.1, so 6.5 = 110.1 in base 2.
- Normalize so there is a single 1 before the point: 110.1 = 1.101 × 2^2. The significand is 1.101 and the true exponent is 2.
- Drop the hidden leading 1. The stored 23-bit fraction is 101 followed by twenty 0s.
- Bias the exponent: stored exponent = 2 + 127 = 129, which is 10000001 in 8 bits.
- The sign bit is 0 (positive). Glue them: 0 | 10000001 | 10100000000000000000000. That 32-bit pattern is exactly 6.5 — no rounding, because 6.5 lands on a grid point.
Try the same recipe on a number like 0.1 and step 1 never terminates: 0.1 in binary is 0.000110011001100... repeating forever, so step 3 must chop it off at 23 (or 52) bits. What gets stored is the nearest grid point, which is microscopically larger than 0.1. The size of that unavoidable snap — the gap between neighbouring grid points relative to the number — is so important it gets its own name, machine epsilon, and the entire next guide. It is the unit in which all of floating point's honesty is measured.
The edges of the grid: zero, subnormals, infinity, NaN
Two exponent values are reserved as escape hatches, and they make IEEE 754 closed and self-consistent rather than full of holes. When the exponent bits are all zeros, the hidden leading 1 is switched off; these subnormal numbers let the grid fade smoothly down toward zero instead of leaving a sudden gap just below the smallest normal number — a property called gradual underflow. Plain 0.0 is the smallest of these (and there is a -0.0 too, which usually behaves like 0.0).
When the exponent bits are all ones, you get the special values. A finite result too big for the grid becomes Inf (so 1.0/0.0 gives +Inf rather than crashing), and a genuinely undefined operation like 0.0/0.0 or Inf - Inf gives NaN, 'Not a Number'. NaN is contagious — almost any arithmetic touching it returns NaN — and it is the only value not equal to itself, so the idiom 'x != x' tests for it. These cases, together with their cousins overflow and underflow, get full treatment in guide 5; for now just know the grid has well-defined edges, not cliffs.
Why this matters for everything ahead
Storing a number is only step one; the standard also specifies how operations behave. Each basic operation (+, -, ×, ÷, sqrt) is defined to compute the exact mathematical result and then round once to the nearest representable number. This is the 'rounded-operation model', and it is wonderfully clean — yet it has a startling consequence: floating-point addition is not associative. Add a tiny number to a huge one and the tiny one may fall off the bottom of the grid and vanish; do the additions in a different order and the tiny contributions survive. So (a + b) + c can differ from a + (b + c) by real, measurable amounts.
That is not a hardware bug; it is the price of a finite grid, and good numerical methods are designed around it. The most dramatic example is when you subtract two nearly-equal numbers: their matching leading digits cancel, leaving only the noisy trailing bits — catastrophic cancellation, which can wipe out most of your ~16 digits in a single subtraction. And summing millions of values naively lets rounding errors pile up, which is exactly what Kahan summation cleverly fixes by carrying a running correction term. Both ideas are coming later in this rung; everything they do is a response to the grid you just learned to read.