Distributed File Systems and Caching

A folder that lives on another machine

In the file-system rung you learned that a path like /home/jo/notes.txt is resolved component by component down a tree of directories and inodes, and that mounting can splice another device into that tree at a mount point. A distributed file system takes that last idea and stretches it across the network: it mounts a folder that does not live on any disk you own at all. After the mount, /shared on your laptop is really a directory sitting on a server in another room — or another building — yet your programs call the same old open(), read(fd, buf, n), and write() against it, blissfully unaware. The most famous example is NFS (the Network File System), and its whole design goal was exactly that invisibility.

How is the trick pulled off? With exactly the machinery from guide 2. When your program calls read() on a file under /shared, the kernel notices the path crosses an NFS mount and, instead of touching a local disk, it packages the request — "read 4 KB at offset 8192 of file handle 0x9a3f" — into a remote procedure call and ships it to the server. The arguments are flattened by marshalling into bytes on the wire, the server does a real local read, and the bytes travel back. The single most important piece of plumbing that makes this clean is the virtual file system layer: the kernel already routes every open()/read()/write() through an abstract interface, so NFS is just another implementation behind it, indistinguishable to your program from the local ext4 driver.

Why we cache — and the bill it quietly runs up

Sending an RPC over the network for every single read is brutally slow. A local disk read might take a fraction of a millisecond; a round trip to a server across a busy network can take many times longer, and you might issue thousands of reads to scroll through one file. The obvious fix is the same one your CPU and your local file system already use everywhere: keep a copy of recently-used data close by. This is caching. The client machine holds a chunk of the remote file in its own memory — its buffer cache — so the second, third, and thousandth read of those bytes are served locally at memory speed, and the server is never even told. For read-heavy workloads this can turn a sluggish share into something that feels exactly like local disk.

But here is the bill, and it is the central tension of this whole guide. The instant a second client also caches that file, the single true file has fractured into several private copies. Now client A edits a line and stores it in its own cache; client B is still reading its own stale copy and has no idea anything changed. Both believe they hold "the" file, and they disagree. This is the cache consistency problem, and it is just file-sharing semantics from the local file-system rung — but cruelly amplified, because the copies are now separated by a network with delay and no shared clock. Caching did not create the disagreement; it just gave every machine a private place to disagree from.

How NFS actually keeps caches honest

So how does a real system stop the copies from drifting apart? The cheapest, oldest approach NFS used is timeout-based validation, sometimes called close-to-open consistency. The idea: do not try to be perfect, just be cheap and good enough. The client trusts its cached copy for a short window — a few seconds — and only after that window expires does it ask the server "has this file changed since I cached it?" by checking the file's last-modified time. If the server's timestamp is newer, the client throws away its stale copy and refetches; if not, it keeps using the cache. This is a deliberate trade: you accept a small window of staleness in exchange for not flooding the network with validation messages.

1. A program opens a file under /shared. The client checks its cache: do I already hold these bytes, and is my validity timer still running?
2. Cache hit, timer fresh: serve the bytes from local memory immediately. No network message at all — this is the fast path that makes caching worth it.
3. Cache hit but timer expired: send one small RPC asking the server for the file's current last-modified time, and compare it to the time stamped on the cached copy.
4. Times match: the copy is still good — reset the timer and serve locally. Times differ, or it was a cache miss: fetch the fresh bytes from the server over RPC and cache them.

Be honest about what this buys and what it costs. The benefit is huge speed and a simple, stateless server that does not have to remember who is caching what. The cost is that NFS is not perfectly consistent: in that few-second window, two clients really can see different contents of the same file, and the rule is only that a client opening a file after another client has closed it will see that other client's writes — not that simultaneous editors stay in sync. A stronger scheme, used in protocols like AFS and SMB, is a callback or lease: the server promises to actively notify (or revoke a lease from) every client caching a file the moment someone writes it. That gives much tighter consistency, but it makes the server stateful — it must track every outstanding cache — and that bookkeeping becomes a nightmare exactly when a client crashes silently and never releases its lease.

Writes, failures, and where CAP starts to bite

Reading is the easy half. Writing forces a second, sharper decision: when a program calls write(), do you ship the change to the server right now, or hold it in the cache and send it later? Sending immediately is write-through — safe, because the server's copy is always current, but slow, because every write pays a network round trip. Holding it locally and flushing in a batch is write-back — fast, because many writes coalesce into one transfer, but dangerous, because if the client crashes before the flush, those bytes that your program believed were saved simply vanish. This is the very same write-through-versus-write-back dilemma your CPU caches and the local buffer cache face, but with the stakes raised: the gap is now a whole network, and the failure is now another machine you cannot inspect.

Now connect this to the deepest idea in the whole rung, the one guide 5 will make formal. Suppose the network between client and server breaks — a network partition. The client has a write sitting in its cache and a user waiting. It faces a genuine fork in the road. It can refuse to proceed until it can reach the server and confirm the write landed, which keeps everyone's view consistent but makes the file unavailable while the network is down. Or it can keep accepting reads and writes from its local cache to stay available, at the price that its copy and the server's copy are now silently diverging. That is the CAP theorem in miniature: when a partition splits the system, you must choose between consistency and availability — you cannot have both at once. A distributed file system is not exempt; it just usually chooses availability and a weaker consistency, and hopes you do not notice.

One more honest corner, because the network turns a friendly assumption into a trap. On a local disk, a write that the kernel acknowledged is durable enough for most purposes, and an operation like read() can be safely retried by hand if you get an error. Across the network, retrying is subtler. If the client sends "append this line" and the reply is lost, did the append happen or not? Retrying blindly might append the line twice. This is why robust remote operations are designed to be idempotent — written so that doing them once and doing them five times leave the file in the same state — for instance "write these exact bytes at offset 8192" rather than "append at the end." The same lost-reply ambiguity you met with RPC in guide 2 is, at bottom, why a distributed file system has to think so carefully about what its operations even mean.

The shape of the whole picture

        CLIENT  (your laptop)                 SERVER  (in another room)
  +---------------------------+          +----------------------------+
  | program: read(fd,buf,n)   |          |  real local file system    |
  |        |                  |          |     ext4 / inodes / disk   |
  |   virtual file system     |          |            ^               |
  |        |                  |          |            |  real read    |
  |   NFS client + CACHE  <----+--hit-----+            |               |
  |        | (miss / stale)    |          |    NFS server (the truth)  |
  +--------|------------------+           +------------|---------------+
           |   RPC request  ============NETWORK=======>|
           |   (marshalled bytes)  <===== reply ========|

   cache HIT + timer fresh  ->  served locally, ZERO network messages
   cache MISS or expired    ->  one RPC round trip to the server
   network down (partition) ->  must choose: consistency OR availability

Step back and see the recurring shape. A distributed file system is the file abstraction you already love, served over the RPC machinery of guide 2, sped up by a cache that immediately reintroduces the disagreement problem, and patched with a consistency scheme that is always a compromise. Every layer is a familiar idea from a lower rung — virtual file system, caching, mounting, file-sharing semantics — re-encountered across a network where, as guide 1 warned, there is no shared clock and no shared memory and failure is partial. Nothing here is new physics; it is the old ideas meeting their hardest test.

And it sets up the next two guides exactly. Notice how often the answer hinged on a question of order and time: "is my cached copy older than the server's?", "did this write happen before or after that one?". Without a shared clock, even comparing two timestamps from two machines is unreliable. Guide 4 confronts that head-on with logical clocks and the happened-before relation, giving us a principled way to order events without trusting any wall clock. Guide 5 then takes the consistency-versus-availability fork we glimpsed here and makes it a theorem, building real fault tolerance out of consensus and replication. The file system was the concrete place to feel the pain; those guides are the cure.