host-device data transfer
Imagine hiring a thousand-person work crew that lives on a remote island. They can do astonishing amounts of work in parallel — but every scrap of material they need has to be shipped over by boat first, and every finished product has to be shipped back. If the job is huge, the boat trips are a small overhead; if the job is tiny, you spend more time loading and unloading boats than working. Host-device data transfer is those boat trips: a GPU (the device) has its own separate memory, and before it can compute, the data must be copied from the CPU's memory (the host) across the connection between them, then results copied back afterward.
Concretely, the CPU and GPU usually have physically separate memories connected by a bus such as PCI Express. A typical GPU job has four phases: allocate memory on the device, copy the input data host-to-device, launch the kernel that computes, then copy the results device-to-host. That bus has real, finite bandwidth — far lower than the GPU's own internal memory bandwidth — and each transfer also has a fixed startup latency, so many tiny transfers are far worse than one big one. This is why the data-movement cost, not the arithmetic, is often what limits real GPU programs.
The honest lesson every GPU beginner learns the hard way: the transfer can erase the speedup. A computation that runs 50 times faster on the GPU is no win if copying the data over and back takes longer than the whole thing would have on the CPU — for small or transfer-heavy problems the GPU is a net loss. The standard cures are to move data once and keep it resident on the device across many kernels (amortize the trip), to overlap transfers with computation so the boat sails while the crew works, and to do enough arithmetic per byte transferred (high arithmetic intensity) that the trip pays for itself. Always count the transfer when you judge whether offloading to a device is worth it; ignoring it is the classic beginner mistake.
Square 10,000 numbers on the GPU: the squaring is trivially fast, but you must copy 40 KB over and 40 KB back. The copy plus its startup latency can dwarf the computation, so this tiny job is slower on the GPU than the CPU. The same kernel on 10,000 dense matrices, where the data is reused heavily, easily wins because the transfer is amortized over enormous arithmetic.
The bus between host and device is the bottleneck no kernel can dodge; count the transfer before deciding to offload.
A GPU speedup measured without the data transfer is a fiction. For small or transfer-heavy problems the host-device copy can cost more than the computation, making the GPU a net loss; keep data resident on the device and overlap transfers to amortize it.