Persistent Memory and New Storage Models

The wall that always stood between memory and storage

Step back and notice an assumption so deep you have never questioned it. Everything in your program lived in one of two worlds. In the volatile world you reach data with a pointer: you write *p = 42 and the CPU stores a value into byte-addressable memory, at full speed, but the instant the power blinks the value is gone. In the durable world you reach data through a file descriptor: you call write() and read(), the bytes survive a reboot, but you pay a system-call crossing and the data moves in blocks of 512 or 4096 bytes, never one byte at a time. Two address modes, two speed classes, two completely different programming styles — and a wall between them you have always just accepted.

That wall is not an accident; it falls right out of the memory hierarchy you met earlier. Closer to the CPU, storage is small, fast, and volatile (registers, caches, DRAM); further away it is large, slow, and durable (SSD, disk). Every layer is a trade between speed and permanence, and the dividing line between volatile and durable has, for decades, sat exactly between DRAM and the disk. Persistent memory (often written pmem, and sold as devices like Intel Optane DC) is a genuinely new layer that lands in the gap: it plugs into the memory bus like DRAM, the CPU addresses it with ordinary loads and stores, it is only a few times slower than DRAM — and it keeps its contents when the power goes away. For the first time, durable storage is on the wrong side of the wall.

What changes when a store can survive a crash

Picture the simplest possible use. You take a region of persistent memory, hand it to mmap() so it appears in your process's address space, and now you hold a plain pointer into durable storage. Updating a record on disk used to mean open(), seek to an offset, write() a block, and worry about buffering; now it is one line — node->next = newp — a single store instruction that updates a durable data structure in place. No serialization, no marshalling a struct into a byte buffer, no copying through the kernel. You can build a linked list, a B-tree, a hash table directly in persistent memory and traverse it with ordinary pointer chasing, and it is still there after you pull the plug. That is the dream, and it is real.

Now the hard truth that makes this a frontier rather than a free lunch. Your store instruction does not write straight to the persistent media. It writes to the cache line, where it may sit in volatile CPU cache for an unbounded time before being evicted to the pmem. So between "the store executed" and "the data is actually durable" there is a gap — and if the power fails in that gap, the value is lost exactly as if it had never been written. Worse, the cache and the memory controller are free to write back dirty lines in any order they like, which is the same reordering freedom you met in the synchronization rung. So a crash can leave you with the second half of an update persisted and the first half still volatile and now gone — a torn, half-finished structure with a dangling pointer baked permanently into durable memory.

The flush-and-fence dance, spelled out

Because the CPU will not push data to the media on its own, making a store durable is an explicit, three-move sequence you perform by hand. First do the store, writing your bytes into the cache. Second, flush that cache line toward persistence with an instruction such as clwb (cache-line write-back) — on x86 it tells the hardware to send the dirty line down toward the memory controller. Third, issue a store fence (sfence) so that all those flushes are guaranteed complete before the program continues; without it, the very next store you do could reach durability before the one you just flushed. Get the order wrong, skip the fence, and you have written code that is correct on every test run and silently corrupts on the one crash that matters.

Making one store durable on persistent memory (x86 sketch):

    node->value = 42;        // 1. store: lands in the volatile cache line
    _mm_clwb(&node->value);  // 2. flush that line toward the pmem media
    _mm_sfence();            // 3. fence: don't proceed until the flush retires
    // only NOW is node->value guaranteed durable across a power cut

Contrast with the classic durable-write path through a file:

    write(fd, buf, n);       // bytes copied into the kernel page cache
    fsync(fd);              // force page cache + device cache to the disk
    // fsync() is the heavyweight syscall equivalent of clwb + sfence

It is worth seeing this as the direct, instruction-level cousin of something you already know. When you call fsync() on a file descriptor, you are asking the kernel to force the page cache and the device's own cache all the way down to durable media — the heavyweight, syscall-priced version of "make my writes survive a crash." The clwb-plus-sfence dance is that exact same idea pushed down to the bare metal: same goal, no kernel crossing, single-cache-line granularity. Understanding pmem is largely understanding that flushing for durability is now your explicit job at the speed of an instruction, not something a filesystem call quietly handles for you in 4 KiB blocks.

Crash consistency: keeping a structure whole

Making one store durable is the easy part. The real problem is making a multi-store update durable as a unit — because a power cut can land between any two of your stores. Suppose you insert into a sorted list: you write the new node's fields, then flip the predecessor's next pointer to point at it. If the crash lands after the pointer flip but before the new node's contents were flushed, durable memory now holds a pointer into a node full of garbage. This is the same crash consistency problem the file system has always faced, except the file system solved it for you behind fsync(); on raw pmem the burden is back on the programmer, at the granularity of individual cache lines.

1. Decide the update's invariant: the one property that must hold even right after a crash — here, "any reachable next pointer points at a fully-initialized node."
2. Write all the new node's fields first, then flush and fence them, so the node's contents are guaranteed durable before anything points to it.
3. Only now perform the single linking store (the predecessor's next pointer) that makes the node reachable — this one store is the atomic commit point.
4. Flush and fence that linking store too, so the commit itself is durable; a crash before it leaves the old list intact, a crash after it leaves the new node fully present.

That discipline — order your stores so that a single, final, durable store is the moment of commit — is exactly the idea behind a write-ahead log and behind every journaling file system, now applied byte by byte. When even a single commit store is too big to be atomic (it spans two cache lines, say), real systems fall back to logging: write the intended change to a small durable redo log, flush it, then apply it in place, so a crash mid-apply can be finished by replaying the log on restart. This is why nobody sane writes raw clwb-and-sfence code by hand for anything serious. Libraries like Intel's PMDK give you transactions over pmem — you mark a block of stores as one atomic unit and the library does the logging, flushing, fencing, and crash-time recovery for you — turning the frontier back into something an ordinary programmer can use safely.

The other new storage models, and where this is heading

Persistent memory is the most radical shift, but it is one of several places the old storage assumptions are cracking. Today's SSD already lies to you: it pretends to be a simple array of fixed blocks, while a hidden Flash Translation Layer remaps every write to spread wear and runs garbage collection underneath, which is why an SSD needs the TRIM command to be told which blocks are truly free. Newer interfaces strip that fiction away — zoned namespaces and open-channel SSDs expose the real flash geometry so the file system, not firmware, decides placement. And CXL, a cache-coherent interconnect, lets a machine attach pools of memory (including persistent memory) over a fast link and even share them across servers, blurring the line between "my RAM" and "the storage tier" even further.

Pull these threads together and the lasting lesson is about a leaky abstraction finally tearing. For decades the clean split — pointers for fast volatile memory, file descriptors for slow durable storage — was a useful simplification, and the mmap() trick of mapping a file into memory was the closest the old world came to bridging it (but a page fault still pulled data from disk underneath). Persistent memory removes the disk underneath entirely, and with it goes the comfortable lie that durability is something far away and coarse-grained that the kernel handles for you. What remains is a sharper, more honest model: durability is a property you reason about per store, you pay for it with explicit flushes and fences, and you keep your structures consistent with the same logging discipline databases and file systems have used all along — now in your own address space.