A different kind of cargo
Everything you have carried over the network so far — a web page, an email, a downloaded file — shares one forgiving property: it does not care exactly when each byte arrives, only that every byte eventually does, intact and in order. That is precisely the deal best-effort IP plus TCP is built to honor. This rung is about a fundamentally different cargo: continuous media, the sound and pictures of a phone call, a live game stream, or a video meeting. Here the timing is the content. A frame that arrives correctly but three seconds late is, for a live call, simply garbage.
Multimedia networking is the study of how we coax this time-sensitive cargo across a network that makes no timing promises at all. The tension is the whole story. The Internet's core was kept deliberately dumb and simple — that is what let it scale, as you saw far back in the foundations rung — so it offers no built-in notion of "this packet is urgent." Real-time media has to be built on top of that indifference. The rest of this guide names the three things that can go wrong, the one trick that saves us most of the time, and the heavier machinery for when that trick is not enough.
Three classes, three different appetites
Not all media is equally fussy, and the differences decide which tricks help. The first class is streaming stored media: a recorded video or podcast you press play on. The content already exists in full on a server, so the player can quietly download a few seconds ahead and stockpile them before showing anything. That stockpile — a playout buffer — is what lets streaming stored media ride out a rough patch in the network without you noticing. The only real cost is a short startup delay, which we happily trade for smoothness.
The second class is live streaming: a sports match or concert sent out as it happens. It is like stored media in that the flow is one-way and the player can still buffer a little, but it cannot buffer far ahead — there is nothing recorded past "now." A live stream tolerates a few seconds of delay (you have all seen a broadcast lag behind the room next door), and it uses that delay budget to smooth things over, but the budget is tight.
The third class is the hardest and most interesting: conversational real-time media — a VoIP call or a video meeting, where two people are talking back and forth. Here you cannot pre-buffer, because the next thing said does not exist until the other person says it, and you cannot hide behind a long delay, because conversation breaks down once the round-trip lag passes roughly a few hundred milliseconds — people start talking over each other. Conversational real-time media has the tightest timing budget of all, which is why it sits at the center of this whole rung.
The three enemies: delay, loss, and the sneaky one
Three things can hurt real-time media, and it is worth keeping them strictly apart. The first is delay (latency): how long a packet takes to travel end to end. Recall from the foundations rung that delay is mostly fixed physics — propagation across distance plus a little processing — and crucially, more bandwidth does not shrink it. A fatter pipe lets more packets flow per second; it does not make any one packet travel faster than the speed of light. For conversation, total round-trip delay must stay small or talking turns awkward.
The second enemy is packet loss. On a best-effort net, a congested router with a full queue simply drops packets, and lost audio or video samples cannot be re-sent in time to help — by the time a replacement arrives, that instant of sound is long past. Surprisingly, media tolerates a little loss far better than you would guess. A few missing milliseconds of audio can be masked by guessing what was probably there, a technique called loss concealment that guide 2 will look at. So loss is bad, but a small amount is survivable, which is exactly why we often refuse to pay TCP's full price to eliminate it.
The third enemy is the sneaky one, and it is the true villain of real-time media: jitter. Jitter is the variation in delay from packet to packet. Picture audio sent as one little packet every 20 milliseconds, a steady drumbeat. Each packet waits a different amount of time in the queues of routers along the way, so they arrive unevenly — one early, the next late, two bunched together. Even if the average delay is fine, that unevenness is poison for media that must be replayed at a perfectly steady rate. Jitter is why a call sounds choppy even when nothing was actually lost.
The playout buffer: trading a little delay for a lot of smoothness
Here is the single most important idea in the whole rung, and the good news is it is cheap and it works without any help from the network. The receiver does not play each packet the instant it arrives. Instead it holds incoming packets briefly in a playout buffer — a small reservoir — and plays them out on a smooth, fixed clock. Early packets wait in the reservoir; the late ones get a head start built in. As long as a packet's extra delay stays inside the cushion you set aside, the listener hears a perfectly even stream. You have absorbed the jitter by adding a small, constant delay on purpose.
Packets leave the sender on a perfect 20 ms beat:
sent: |--20--|--20--|--20--|--20--|
Network jitter scatters their arrival times:
arrive: | . ..|. .| . . | .. .| (uneven!)
Receiver holds them in a buffer, then plays on a steady clock:
played: |--20--|--20--|--20--|--20--| (smooth again)
^ small fixed startup delay = the jitter cushionBut the buffer cannot be free, because it is itself made of delay, and we just said delay is one of the enemies. Make the cushion too small and any late packet misses its slot and is lost; make it too large and a conversation drifts into that awkward, talk-over-each-other zone. So the playout buffer is a tuning problem: just enough delay to swallow the jitter, no more. For a movie you can be generous; for a live call you must be stingy. Guide 2 is devoted entirely to choosing that size well, including how a smart receiver re-estimates it on the fly as network conditions shift.
The protocols, and asking the network for favors
Media still needs packaging and coordination, so a small family of protocols grew up to ride on top of UDP (chosen, you will recall, precisely because TCP's retransmissions arrive too late to help). The first is RTP, the Real-time Transport Protocol: it wraps each chunk of audio or video with a sequence number (to detect loss and reordering) and a timestamp (so the receiver can drive its playout clock). Riding alongside it, RTCP carries periodic quality reports back to the sender — how much loss, how much jitter — so the sender can adapt. Separately, SIP is the signaling protocol that finds the other party and sets up a call, the way you would dial before talking. Guide 3 walks an entire VoIP call through these.
Protocols and playout buffers are end-host tricks: clever, but they cannot conjure delay out of thin air if the network in the middle is genuinely overloaded. So the deeper question of this rung is whether we can ask the network itself for special treatment — to make routers play favorites for time-critical traffic. That whole idea has a name, quality of service (QoS): mechanisms by which a network gives some flows better delay, loss, or bandwidth than others, rather than treating every packet identically.
QoS rests on two kinds of machinery, both previewed here and detailed in guide 4. Inside a router, scheduling disciplines decide whose packet leaves the queue next: plain first-in-first-out (FIFO) treats everyone equally; a priority scheme lets urgent voice jump the line; weighted fair queuing hands each flow a guaranteed slice of the link. At the network's edge, policing and shaping control how fast a flow may inject traffic — policing drops or marks packets that exceed an agreed rate, while shaping buffers them to smooth bursts — and both are usually built from two simple models, the leaky bucket and the token bucket.
Why the Internet mostly stays best-effort anyway
Given all that machinery, you might expect the Internet to be full of guaranteed-quality lanes. It is not, and the reason is the quiet theme of this rung. Two grand architectures were designed to deliver QoS across the whole Internet. IntServ asks every router on a path to reserve resources for each individual flow before it starts — a per-call guarantee, like booking a private lane. It works, but it does not scale: a backbone router carrying millions of flows cannot track state for each one. DiffServ is the lighter answer: do not reserve per flow; instead mark each packet with a small class tag at the edge and have routers give whole classes rough preferential treatment. Guide 5 compares them in full.
Yet even DiffServ is patchy in practice, and across the public Internet most traffic still travels best-effort — every packet treated the same, no promises. Why did the simpler, dumber option win? Partly because guarantees are hard to enforce across the many independent networks a packet crosses, each with its own owner and interests. But mostly because the end hosts got so good at coping: generous playout buffers, adaptive codecs, adaptive bitrate streaming that lowers video quality when the network tightens, and ever-fatter links that keep average congestion low. When the edges are clever and capacity is cheap, the temptation to re-engineer the core fades.