The same delay every time would be fine
The previous guide named real-time media's three enemies — delay, loss, and the sneaky one, jitter — and promised this guide would unpack the cure. Let's start by clearing up something that surprises most people: a phone call can survive a large delay perfectly well, as long as that delay is the same for every packet. Imagine the network always added exactly 200 milliseconds to every voice packet. Annoying for conversational media, where you'd start talking over each other, but the audio itself would play back perfectly smoothly — each chunk arrives 200 ms late, in order, evenly spaced, just like it was spoken.
The trouble is that the Internet does not give you the same delay every time. A voice source chops sound into a steady stream of little packets — say one every 20 ms — and hands them to the network evenly spaced, like a metronome. But each packet then takes its own journey through routers whose queues are filling and draining moment to moment. One packet sails through nearly empty queues; the next gets stuck behind a burst of someone else's traffic. They were sent like clockwork, but they arrive in clumps and gaps. That variation in arrival timing is exactly what we mean by jitter.
Where jitter actually comes from
It helps to break a packet's total delay into pieces. There's the fixed part — propagation delay, the time light or electricity needs to physically cross the wires, which depends only on distance and never changes for a given path. And there's the variable part — queuing delay, the time a packet spends waiting in router buffers behind other packets. Propagation is constant; queuing is not, because how many packets sit ahead of yours depends on what every other user is doing at that exact instant. Jitter is essentially the variation in that queuing component, packet to packet.
Why jitter wrecks playback
Suppose your audio player did the naive thing: play each packet the instant it arrives. The source sent chunk 1 at time 0, chunk 2 at 20 ms, chunk 3 at 40 ms — perfectly even. But say chunk 1 arrives with 30 ms of delay, chunk 2 with 30 ms again, and chunk 3 gets stuck and arrives with 75 ms of delay. Now there's a gap: after chunk 2 finishes, chunk 3 isn't here yet, so the speaker falls silent for a beat, then chunk 3 blurts out late and the next few rush to catch up. The result is the choppy, robotic, gurgling sound everyone recognizes from a bad call. The audio was fine; the timing of delivery destroyed it.
Here is the key insight, and it's almost paradoxical: the cure for variable delay is to add a little fixed delay on purpose. If the receiver refuses to play any chunk the instant it arrives, and instead holds each one for a brief, deliberate wait before playing it, then the slow-arriving packets get time to catch up to the fast ones. We trade a small, constant chunk of latency for a smooth, evenly-paced output. That deliberate waiting room is the playout buffer, sometimes called the jitter buffer.
How the playout buffer works
The trick rests on one piece of information: each packet carries a timestamp saying when it was generated at the source. (We'll see in the next guide that this is exactly what RTP's header provides.) Using timestamps, the receiver picks a single playout point for the whole stream: a fixed offset, call it d, after a packet's generation time. The rule is then simply: a packet generated at time t is played at time t + d, no sooner and no later. Packets that arrive before their playout moment wait in the buffer; the playout point d is chosen large enough that almost all packets make it in time.
- A packet arrives. The receiver reads its source timestamp t and notes when it actually showed up.
- If the local clock has not yet reached t + d, the packet is early — drop it into the buffer to wait, in timestamp order.
- At time t + d, pull that packet out of the buffer and hand it to the audio decoder to play, exactly on its scheduled beat.
- If a packet still hasn't arrived by its t + d moment, it's effectively lost for playback — skip it and use loss concealment to fill the gap, because waiting longer would stall the whole stream.
Because output is now clocked off the source timestamps rather than off arrival times, the player emits one chunk every 20 ms like a metronome again, regardless of how lumpy the arrivals were. The buffer has absorbed the variation. Run the earlier example through it: pick d = 80 ms after generation. Chunk 3 arrived 75 ms late, but its playout moment was 80 ms out, so it was already waiting in the buffer with 5 ms to spare — no gap, no glitch.
The unavoidable trade-off, and a missing packet
Choosing the playout offset d is a genuine tug-of-war, and there is no setting that wins both ways. Make d large and the buffer can soak up even badly delayed packets, so almost nothing is dropped and audio stays smooth — but every word the other person says now reaches your ear that much later, and on a live conversation too much delay makes people talk over each other. Make d small and the conversation feels snappy and responsive — but any packet whose queuing delay spikes past d misses its slot entirely and is treated as lost. Smoothness versus responsiveness: you can favor one only by giving up some of the other.
Good softphones don't pick d once and freeze it; they run an adaptive playout buffer. They continuously measure recent jitter — how spread out arrival delays have been lately — and nudge d up when the network gets choppy, down when it calms. They sneak these adjustments into the natural silences between words, so you never hear the buffer stretching or shrinking. This is why VoIP on a flaky connection sometimes feels like it's drifting a beat further behind: the buffer just grew to ride out a rough patch.
And the packet that genuinely never arrives in time? You don't stall the stream waiting — that would replace one glitch with a worse freeze. Instead you skip it and lean on loss concealment: the decoder hides the hole, for instance by repeating the previous sound snippet or smoothly interpolating across the gap. A single missing 20 ms of speech, papered over this way, is usually imperceptible. This is also why real-time media so often rides on UDP rather than TCP: TCP would dutifully retransmit the lost packet, but by the time the retransmission arrived its playout moment would be long gone — late audio is useless audio, so it's better to skip and conceal than to wait and stall.