Hugepages: The Hidden TLB Tax on LLM Loads

Your CPU has a tiny cache that nobody talks about at parties, and when it overflows your fancy GPU box quietly turns into a very expensive page-table walker. That cache is the TLB, and Linux hugepages are the lever that changes how much memory it can cover. This post shows how to measure TLB misses with perf, compare 4K pages, transparent hugepages (THP) and explicit hugetlb pages, and decide when each one helps or hurts GPU and LLM workloads.

What the TLB tax actually is

Every memory access uses a virtual address, and the CPU has to translate it to a physical one. The translation lookaside buffer (TLB) caches recent translations. On a miss the hardware walks the page tables, which costs extra memory accesses. The kernel's memory-management concepts documentation puts it plainly: the TLB is a scarce resource, applications with large memory working sets will experience performance degradation from TLB misses, and huge pages reduce TLB pressure and improve the hit rate.

The kernel THP documentation gives the two headline benefits: fewer page faults for the same memory, and a single TLB entry that maps a much larger amount of virtual memory, which reduces TLB misses. On virtualised hosts it adds that the TLB can only map the larger size when both KVM and the Linux guest use hugepages, though a significant speedup already happens if only one of them does, because the miss runs faster.

Why does this matter for AI infrastructure? Loading a multi-gigabyte model, streaming a KV cache, or filling pinned host buffers for GPU transfers all touch huge contiguous regions of memory. With 4 KiB pages that is hundreds of thousands of translations per gigabyte. With 2 MiB pages it is 512 times fewer.

Step 1: see what your system does today

# THP policy: [always] madvise never
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

# How much anonymous memory is currently backed by THP
grep -E 'AnonHugePages|HugePages_|Hugepagesize' /proc/meminfo

# Is a specific process using THP? (replace PID)
grep AnonHugePages /proc/$PID/smaps_rollup

Per the kernel docs, the enabled file accepts always, madvise and never. With madvise, THP is only used inside regions an application marked with madvise(MADV_HUGEPAGE). Many distributions ship one of the first two, so check rather than assume. In meminfo, AnonHugePages is THP, while the HugePages_Total, HugePages_Free and Hugepagesize fields describe the explicit hugetlb pool, which is a separate mechanism.

Step 2: measure dTLB misses with perf

Do not turn on hugepages because a blog said so. Measure. The generic perf events dTLB-loads and dTLB-load-misses work on most x86 and Arm machines, though a few virtual machines do not expose them. The test program below walks a big buffer with a page-sized stride so that nearly every access lands on a new page, which is the worst case for the TLB.

// gcc -O2 -o tlbtest tlbtest.c
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <time.h>

int main(int argc, char **argv) {
    size_t size = 4UL << 30;               // 4 GiB
    int use_madvise = argc > 1 && !strcmp(argv[1], "madvise");
    int use_hugetlb = argc > 1 && !strcmp(argv[1], "hugetlb");

    int flags = MAP_PRIVATE | MAP_ANONYMOUS;
    if (use_hugetlb) flags |= MAP_HUGETLB;  // needs vm.nr_hugepages
    char *buf = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1, 0);
    if (buf == MAP_FAILED) { perror("mmap"); return 1; }
    if (use_madvise && madvise(buf, size, MADV_HUGEPAGE)) perror("madvise");

    memset(buf, 1, size);                   // fault everything in

    struct timespec t0, t1;
    clock_gettime(CLOCK_MONOTONIC, &t0);
    unsigned long sum = 0;
    for (int pass = 0; pass < 20; pass++)
        for (size_t i = 0; i < size; i += 4096 + 64)
            sum += buf[i];
    clock_gettime(CLOCK_MONOTONIC, &t1);

    printf("sum=%lu time=%.3fs\n", sum,
           (t1.tv_sec - t0.tv_sec) + (t1.tv_nsec - t0.tv_nsec) / 1e9);
    return 0;
}

# 4K baseline: THP off for this experiment
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
perf stat -e dTLB-loads,dTLB-load-misses,page-faults ./tlbtest

# THP via madvise
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
perf stat -e dTLB-loads,dTLB-load-misses,page-faults ./tlbtest madvise

# Explicit hugetlb: reserve 2048 x 2 MiB = 4 GiB, then map with MAP_HUGETLB
echo 2100 | sudo tee /proc/sys/vm/nr_hugepages
grep HugePages_Total /proc/meminfo
perf stat -e dTLB-loads,dTLB-load-misses,page-faults ./tlbtest hugetlb

Compare the miss ratio (misses divided by loads) and the wall time. Run each case several times and take the median. I am not printing sample numbers: they depend on your CPU's TLB sizes, memory speed and virtualisation layer, and a laptop result would mislead you about a server. What you should see is the miss ratio collapsing for the madvise and hugetlb runs.

THP versus explicit hugetlb pages

THP is opportunistic: the kernel promotes ranges to huge pages when it can, and the khugepaged daemon can collapse existing small pages later. Explicit hugetlb pages are a reserved pool that applications request deliberately, through MAP_HUGETLB, hugetlbfs or memfd. The hugetlb documentation says persistent pages in the pool are returned to it when freed, and that you can change the pool at runtime by writing to /proc/sys/vm/nr_hugepages.

The catch is fragmentation. The same document says that allocating on the kernel command line with hugepages=N is the most reliable method, because memory has not yet become fragmented. A runtime echo into nr_hugepages on a machine that has been up for weeks may reserve fewer pages than you asked for. Always read HugePages_Total back.

# /etc/default/grub  -> GRUB_CMDLINE_LINUX
#   default_hugepagesz=2M hugepagesz=2M hugepages=4096
# Add 1 GiB pages if your CPU supports them:
#   hugepagesz=1G hugepages=8
sudo update-grub && sudo reboot

# After boot
grep -E 'HugePages_(Total|Free)|Hugepagesize' /proc/meminfo
ls /sys/kernel/mm/hugepages/

When THP makes things worse

The kernel documentation is candid about the trade-offs. With THP enabled system wide, an application may mmap a large region and touch a single byte, and a 2M page may be allocated instead of a 4K page for no good reason. That is why it recommends madvise-only mode, particularly on memory-constrained systems, and says applications that really benefit should call madvise(MADV_HUGEPAGE) on their critical mmapped regions.

Latency is the second cost. The defrag setting controls what happens when a huge page is not immediately free. In always mode, per the docs, an application requesting THP will stall on allocation failure and directly reclaim pages and compact memory to get one immediately. In defer mode it wakes kswapd and kcompactd in the background and lets khugepaged install the THP later. The default is madvise, which enters direct reclaim only for MADV_HUGEPAGE regions. If you serve latency-sensitive requests, a stall inside a page fault is exactly the tail spike you do not want, and the PSI memory counters from our earlier post will show it.

# THP only where applications ask, and do not stall to get it
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Watch for memory stalls while you test
cat /proc/pressure/memory
grep -E 'thp_fault_alloc|thp_fault_fallback|compact_stall' /proc/vmstat

What this means for GPU and LLM boxes

• Model loading: a big weights file mapped or copied into host memory touches gigabytes sequentially. THP via madvise, or a hugetlb pool for software that supports it, reduces both page faults and translation misses. Measure load time and dTLB-load-misses before and after.
• Pinned host buffers for GPU transfers: whether your framework backs them with huge pages depends on the allocator and driver, so verify with AnonHugePages and HugePages_Free during a run rather than assuming.
• CPU-side inference and tokenisation with large working sets: this is where TLB pressure shows up most clearly in perf stat.
• Latency-sensitive serving next to batch jobs: prefer madvise with defer+madvise so a page fault never waits on compaction.
• Virtual machines: remember that both host and guest need huge pages for the full effect.

Takeaway

Hugepages are not a magic flag. They trade memory flexibility and possible allocation stalls for fewer page faults and TLB misses. Check your current policy, measure dTLB-load-misses on a workload that looks like yours, prefer madvise-scoped THP with background defrag for mixed machines, and reserve an explicit pool at boot only for software that will really use it.