Memory Hierarchy: Registers to Storage

Every computation your machine performs lives and dies by the memory hierarchy. The CPU does not fetch data directly from your SSD — it pulls from a cascading ladder of memory types, each offering different trade-offs between speed, capacity, and cost. Understanding this hierarchy is not academic window dressing. It is the difference between code that flies and code that crawls.

When a processor needs to execute an instruction, it expects the operand to be available within a single cycle. DRAM cannot deliver that. Neither can your NVMe drive. The gap between CPU speed and memory speed has widened year over year, and the hierarchy exists precisely because no single memory technology can satisfy both the bandwidth demands and the capacity demands of a modern system simultaneously.

Introduction

When to Use / When Not to Use

The memory hierarchy is not something you “use” in the sense of calling an API. It is a structural reality you work with or against. You exploit it through your code’s locality patterns — how your data accesses cluster in time and space.

When you benefit from it:

• Sequential data access (scanning arrays, reading files) — hardware prefetchers detect stream patterns and stage data before you ask
• Hot data fitting in cache (working sets under a few MB) — cache hit rates approach 99% for well-structured access
• Recursive algorithms with stack-scoped data — L1 cache backs the call stack directly

When it fights you:

• Random access scattered across large address spaces — every miss costs a DRAM round-trip (~100ns)
• Data sets larger than last-level cache (LLC) — thrashing destroys throughput
• Poorly localized access in nested loops (e.g., column-major access to row-major data) — cache line utilization drops below 20%

Architecture or Flow Diagram

The hierarchy is a layered model where each layer serves the layer above it as a faster, smaller backing store.

flowchart TD
    CPU["CPU Core<br/>1-2 ns access"]
    L1["L1 Cache<br/>~1-2 cycles<br/>32-64 KB"]
    L2["L2 Cache<br/>~3-10 cycles<br/>256 KB - 1 MB"]
    L3["L3 Cache<br/>~20-40 cycles<br/>Shared 8-64 MB"]
    DRAM["DRAM<br/>~100 ns<br/>8-64 GB"]
    SSD["NVMe SSD<br/>~100 μs<br/>256 GB - 2 TB"]
    HDD["HDD<br/>~10 ms<br/>1-10 TB"]

    CPU --> L1
    L1 --> L2
    L2 --> L3
    L3 --> DRAM
    DRAM --> SSD
    SSD --> HDD

    subgraph Locality["Temporal & Spatial Locality"]
        L1L2["Data reused within<br/>L1/L2 residency time"]
        DRAML["Data cached from DRAM<br/>before CPU asks"]
    end

Each layer fills itself through a combination of temporal locality (reusing recently accessed data) and spatial locality (prefetching adjacent addresses).

Core Concepts

Registers: The Zero Layer

CPU registers are the only memory the ALU operates on directly. x86-64 has 16 general-purpose registers, each 64 bits wide. ARM has 31 general-purpose registers. Access is sub-nanosecond, and the operand bottleneck is purely in the fetch-decode-execute pipeline — not memory latency.

The compiler’s register allocator is the first layer of the memory hierarchy management. When you declare a variable in C, the compiler decides whether it lives in a register for the duration of a basic block or gets spilled to the stack.

Cache Hierarchy: L1, L2, L3

Modern CPUs use a multi-level cache hierarchy. The Intel Core i7-12700 has:

• L1d: 48 KB per core, ~4 cycle latency
• L2: 512 KB per core, ~12 cycle latency
• L3: 25 MB shared, ~20-40 cycle latency

AMD’s Zen 3 architecture uses similar tiers but with different latency characteristics due to the Infinity Fabric interconnect.

Cache lines are the unit of transfer — typically 64 bytes on x86 and ARM. When the CPU reads one byte from DRAM, the memory controller pulls in the entire cache line containing that address. This is why sequential access to array elements is dramatically faster than strided access — each DRAM fetch services multiple array elements in one shot.

Main Memory: DRAM

Dynamic RAM stores each bit as a charge on a capacitor, requiring periodic refresh (~64 ms interval). The cell design is minimal — one capacitor and one transistor per bit — enabling massive density. But reading requires detecting tiny charge differences, which introduces the ~100 ns read latency.

DDR5 DRAM transfers 64 bytes per cycle at 4800 MT/s, yielding ~384 GB/s theoretical bandwidth on a single channel. Dual-channel configurations double that. Memory bandwidth is often the ceiling for streaming workloads.

Storage Hierarchy: SSD and HDD

NVMe SSDs using NAND flash sit in the hierarchy as persistent storage with DRAM-like interface latency but flash-like write characteristics. NAND flash pages (~16 KB) can only be written after being erased (in blocks of ~256 pages). This erase-before-write behavior is hidden by the SSD controller’s firmware, which manages mapping between logical block addresses and physical flash pages.

HDDs remain relevant for cold storage due to cost per terabyte. The disk’s mechanical nature — rotating platters at 5400-7200 RPM and actuator arm movement — produces seek times measured in milliseconds. Sequential throughput is respectable (~100-200 MB/s), but random I/O is atrocious (~100 IOPS).

Production Failure Scenarios

Cache Thrashing in High-Concurrency Workloads

Failure: Multiple threads accessing data that maps to the same cache set. Cache associativity limits how many unique addresses can be cached simultaneously. When threads fight over the same sets, effective cache hit rates collapse to near zero despite having adequate overall capacity.

Mitigation: Use numactl or libnuma to bind threads to specific NUMA nodes, reducing cross-node memory traffic. Profile with perf stat -e cache-misses,cache-references to identify hot lines. Pad critical data structures to spread them across cache sets.

Row Hammer and Bit Flips

Failure: Repeated activation and deactivation of DRAM rows causes electromagnetic interference that flips bits in adjacent rows. attackers can deliberately trigger row hammer to corrupt memory, bypassing software protections.

Mitigation: Enable Target Row Refresh (TRR) in the DRAM controller. Use memory that supports on-chip ECC. Some workloads require disabling DRAM auto-refresh in favor of software-managed refresh cycles. Cloud providers offer instance types with ECC memory as a standard feature.

SSD Write Amplification

Failure: The SSD controller may write more flash than the host requests due to garbage collection and wear leveling. A 4 KB write can trigger writing multiple 256 KB erase blocks, dramatically accelerating flash wear. In write-heavy workloads, SSDs can fail within months.

Mitigation: Monitor SMART attributes (nvme smart-log /dev/nvme0n1). Use filesystems with discard mount option for thin-provisioned SSDs. Choose enterprise SSDs with higher write endurance ratings (DWPD — drive writes per day). For extreme write workloads, consider persistent memory (Intel Optane) as a write buffer.

Trade-off Table

Implementation Snippets

Cache Line Access Pattern (C)

#include <stdint.h>
#include <time.h>

#define N 1024 * 1024
#define STRIDE 16

// This demonstrates the dramatic performance difference
// between sequential access (cache-friendly) and strided access
double benchmark_array_access(int64_t* arr, int stride, int n) {
    struct timespec start, end;
    clock_gettime(CLOCK_MONOTONIC, &start);

    int64_t sum = 0;
    for (int i = 0; i < n; i += stride) {
        sum += arr[i];  // Sequential: high cache hit rate
    }

    clock_gettime(CLOCK_MONOTONIC, &end);
    double elapsed = (end.tv_sec - start.tv_sec) +
                     (end.tv_nsec - start.tv_nsec) / 1e9;
    return elapsed;
}

// On a modern CPU with 64-byte cache lines:
// Sequential access (stride=1): ~0.5-1 ns per element
// Strided access (stride=16):   ~50-100 ns per element (DRAM round-trip)

Cache Size Detection (Python)

#!/usr/bin/env python3
"""Detect L1, L2, L3 cache sizes on Linux via sysfs."""
import os
import pathlib

def get_cache_size(level: int, cache_type: str = "Size") -> int:
    """Read cache size in bytes from sysfs."""
    base = pathlib.Path(f"/sys/devices/system/cpu/cpu0/cache")
    for entry in base.iterdir():
        if entry.is_dir():
            index_file = entry / "level"
            type_file = entry / "type"
            if index_file.exists() and type_file.exists():
                level_val = int(index_file.read_text().strip())
                type_val = type_file.read_text().strip()
                if level_val == level and type_val == cache_type:
                    size_str = (entry / "size").read_text().strip()
                    # Size is reported as "32 KB" or similar
                    multiplier = 1024 if "KB" in size_str else 1
                    if "MB" in size_str:
                        multiplier = 1024 * 1024
                    return int(size_str.rstrip(" KMGT")) * multiplier
    return 0

# Example output:
# L1 dcache: 32768 bytes (32 KB)
# L2 cache:  524288 bytes (512 KB)
# L3 cache:  16777216 bytes (16 MB)

Observability Checklist

• Cache hit/miss ratios: perf stat -e cache-misses,cache-references ./program
• Last-level cache (LLC) miss rate: Intel PCM, AMD μProf, or Linux perf stat
• Memory bandwidth utilization: intel_pcm_bandwidth or likwid
• DRAM page activation rates: ipmctl for Intel Optane DCPMM
• NVMe SMART health: nvme smart-log /dev/nvme0n1 --namespace-id=1
• Disk I/O latency distribution: iostat -x 1 or blktrace
• Page fault rate: vmstat 1 — in column for interrupts from I/O
• NUMA topology: lscpu or numactl --hardware
• Core to memory controller latency: perf sched latency for scheduling events

Common Pitfalls / Anti-Patterns

Cache Timing Side Channels: Spectre, Meltdown, and subsequent variants exploit the memory hierarchy to leak data across security boundaries. The attack triggers speculative execution to cache a target line, then measures access time to infer whether the line was cached — revealing a bit of secret data. Mitigations include:

• IBRS/STIBP (Indirect Branch Restricted Speculation / Speculative Tracking Indirect Branch Prediction): kernel isolation of branch prediction state
• L1D flush on kernel entry: l1d_flush=on kernel parameter forces L1D cache flush on context switch
• Software mitigations: __attribute__((__target__("arch=zen3"))) for AMD Zen 3 with direct-walk optimization disabled

Memory Disclosure via Cold Boot Attacks: RAM retains data for seconds to minutes after power loss, enabling an attacker with physical access to read sensitive data from DRAM. Mitigations include full-disk encryption (LUKS with TPM unlock) and setting memory wipe on shutdown.

PCIe bus snooping: Memory traffic on the memory bus can be observed via PCIe traffic analysis. Use memory encryption features like AMD SEV (Secure Encrypted Virtualization) for sensitive workloads.

Common Pitfalls / Anti-patterns

Pitfall: Confusing cache associativity with cache size. A 1 MB L2 cache with 8-way set associativity has 8192 sets. If your working set maps to the same set index, you effectively have only 8 cache lines despite 1 MB of total capacity. This is the “cache set collision” problem.

Pitfall: Ignoring write-through cache behavior in multi-core systems. When core 0 writes to a cache line that is shared (read-only or read-write state) with core 1’s L1 cache, the cache coherence protocol invalidates core 1’s copy. Subsequent reads by core 1 cause expensive cache line evictions and re-fetches. This is why false sharing — two unrelated variables on the same cache line, written by different threads — devastates multi-threaded performance.

Pitfall: Using malloc/free on large objects as if they are free. The allocator rounds up to the nearest cache line or page. Allocating and freeing many small objects fragments the heap and increases TLB pressure.

Anti-pattern: Blindly using memset on large arrays. memset touches every byte sequentially, which plays perfectly into hardware prefetching. But if the array is larger than available cache, the write stream saturates memory bandwidth and starves other memory operations.

Quick Recap Checklist

• The memory hierarchy exists because no single memory technology satisfies both speed and capacity requirements simultaneously
• Cache lines (64 bytes) are the fundamental unit of cache transfer — sequential access maximizes line utilization
• L1/L2/L3 caches are SRAM-based, fast, small, and per-core or shared; DRAM is slower, larger, and system-wide
• Temporal locality = reusing recently accessed data; spatial locality = accessing nearby addresses
• NUMA (Non-Uniform Memory Access) means memory attached to distant CPUs has higher latency
• Row hammer exploits electromagnetic interference between DRAM rows to flip bits
• False sharing (unrelated variables on the same cache line, written by different threads) collapses multi-core performance
• perf stat with cache-misses and cache-references events reveals how well your workload utilizes the cache hierarchy
• SSD write amplification can accelerate flash wear beyond specifications — monitor SMART health attributes
• Speculative execution side channels (Spectre/Meltdown) exploit cache timing to leak data across privilege boundaries

Memory Hierarchy Trade-off Analysis

Interview Questions

struct Foo {
    char a;      // 1 byte
    long b;      // 8 bytes
    char c;      // 1 byte
};

Further Reading

CPU Cache Architecture Deep Dive

Modern CPUs implement a hierarchical cache architecture with distinct characteristics:

L1 Cache (Level 1)

• Split into L1 instruction cache (L1I) and L1 data cache (L1D)
• Size: 32-48 KB per core on Intel/AMD
• Latency: ~4 cycles
• Associativity: 8-way set associative typical
• Cache line: 64 bytes

L2 Cache

• Unified (instruction + data)
• Size: 256 KB - 1 MB per core
• Latency: ~12 cycles (Intel Haswell)
• Often inclusive of L1 (meaning L1 lines are also in L2)

L3 Cache (Last Level Cache)

• Shared across all cores on the socket
• Size: 8-64 MB depending on SKU
• Latency: 20-50 cycles
• Usually exclusive (L1/L2 not duplicated here)
• On AMD Zen 2/3: separate CCX-chips means L3 is per-CCX, not unified across socket

Cache Hierarchies in ARM ARM Cortex-A72 (used in AWS Graviton2, Apple A-series): 64-byte lines, 3-level hierarchy with L1 32-48 KB per core, L2 512 KB per core, L3 up to 4 MB cluster-level.

Memory Latency Numbers (2024)

Cache Line Alignment and Performance

Misaligned access across cache line boundaries can cause a single access to trigger two cache line loads. Compiler auto-vectorization (SIMD) works best when data is cache-line-aligned and accessed in sequential patterns.

// Cache-friendly: sequential access, aligned structures
struct packet {
    uint64_t timestamp;
    uint32_t flags;
    uint32_t length;
    uint8_t data[60];  // 64-byte cache line size
} __attribute__((aligned(64)));  // Force cache line alignment

Key Takeaways

• L1/L2/L3 caches are transparent to software but their behavior determines effective memory latency
• Cache lines are 64 bytes on x86/ARM; accessing adjacent data in sequence maximizes line utilization
• Multi-level cache hierarchy reflects physics: larger caches are slower
• NUMA topology means remote DRAM access costs 30-50% more than local
• Row hammer is a real hardware vulnerability affecting DRAM; TRR and ECC provide mitigation

Conclusion

The memory hierarchy is a structural reality every program operates within — registers, caches, DRAM, and storage each offer different trade-offs between speed, capacity, and cost. No single memory technology can satisfy both CPU speed demands and the capacity needs of modern workloads simultaneously.

Sequential data access exploits spatial locality, letting one DRAM fetch service multiple array elements via the 64-byte cache line. Temporal locality means recently accessed data stays close to the CPU in faster, smaller caches. When working sets exceed cache capacity, the result is thrashing — performance collapses as the system spends more time moving data than processing it.

False sharing between threads on the same cache line can degrade multi-core performance by 10-100x. NUMA awareness matters for workloads spanning multiple CPU sockets. Speculative execution side channels (Spectre/Meltdown) exploit the hierarchy to leak data across security boundaries.

For your next step, explore paging and page tables to understand how the OS manages memory at the page level, or virtual memory to see how disk extends physical memory capacity when working sets exceed available RAM.