Kernel Internals & OS Construction

system-call implementation

A program in user mode is deliberately fenced off from the hardware — it cannot touch the disk, the network, or another process's memory directly. So when your code calls something like read or write, how does that request actually cross into the privileged kernel and back? System-call implementation is the end-to-end machinery that carries out that crossing safely. Think of it like ordering at a bank teller window with a thick bulletproof partition: you cannot reach into the vault, so you write your request on a slip, slide it through the slot, the teller (running with full authority) does the work, and slides the result back. The slot, the slip, and the rules for using them are the system-call mechanism.

Concretely, the journey has clear steps. (1) Your code calls a thin user-space wrapper from the C library (for example, glibc's read). (2) The wrapper places the system-call number and the arguments where the kernel expects them, following a strict calling convention defined by the ABI (on x86-64 Linux: the call number in the rax register, arguments in rdi, rsi, rdx, and so on). (3) It executes the special syscall (or trap) instruction, which switches the CPU from user mode to kernel mode and jumps to one fixed kernel entry point. (4) The kernel saves the user registers, uses the call number to index the system-call table and find the right handler, and validates and copies the arguments across the boundary (it never trusts a user pointer — it checks the address is in user space and copies the data in with a safe routine like copy_from_user). (5) The handler does the work, the kernel puts the return value in the agreed register, restores state, and returns to user mode just after the trap instruction.

It matters because this is the only sanctioned door between the two worlds, and its safety rests on never trusting user input: a single missing bounds check on a copied pointer can let a program read or corrupt kernel memory. A common misconception is that a system call is just a function call — it is far heavier, because it involves a mode switch, register save and restore, and validation; this cost is exactly why batching work (read 64 KB at once instead of one byte 65536 times) is so much faster.

You call read(fd, buf, 100). glibc puts the read number into rax and fd, buf, 100 into rdi, rsi, rdx, then runs syscall. The CPU enters kernel mode at the syscall entry; the kernel looks up entry sys_read in the table, checks that buf is a valid user address for 100 bytes, reads from the file, copies bytes into buf with copy_to_user, sets rax to the number of bytes read, and returns. To your program it looked like an ordinary function returning a count.

Wrapper -> set up registers -> trap to kernel -> validate, dispatch, do work -> return.

The cardinal rule is that the kernel must never trust a pointer or length handed in by user code — it must validate every one before dereferencing. Many serious kernel vulnerabilities are exactly a missed check on a user-supplied address or size at this boundary.

Also called
syscall mechanismhow a system call works系統呼叫機制