a thread pool

Imagine a busy restaurant. The naive plan is to hire a brand-new waiter for every single order, walk them through onboarding, let them carry one plate, then fire them. That is absurd: hiring and firing dwarf the actual work. The sane plan is to keep a fixed staff of waiters on the floor; each one grabs the next order off the rail, serves it, and comes back for more. A thread pool is that fixed staff for a program — a small set of long-lived worker threads created once up front, that pull jobs from a shared queue and run them.

Concretely, a thread pool holds N worker threads (often N is around the number of CPU cores) and a task queue. When your code wants something done concurrently, it does not call pthread_create(); it submits a task (a function plus its data) onto the queue. An idle worker dequeues the task, runs it to completion, then loops back to wait for the next one. Creating a thread is expensive — the OS must allocate a stack (commonly several hundred KiB to a few MiB), set up bookkeeping, and the scheduler must learn about it. The pool pays that cost N times at startup instead of once per task, so submitting a task becomes cheap: just an enqueue.

Two things make pools beat thread-per-task. First, amortization: thousands of tasks share the same handful of threads, so per-task overhead is tiny. Second, bounded parallelism: with thread-per-task, ten thousand simultaneous requests would spawn ten thousand threads, exhausting memory and thrashing the scheduler; a pool with, say, 8 workers runs at most 8 tasks at once and the rest wait politely in the queue. The honest caveat: pool size is a real tuning knob. Too small and CPU-bound tasks leave cores idle; size it for CPU work and then let a worker block on I/O, and you can stall the whole pool — see the don't-block-the-executor rule.