Reading Linux PSI: Why Is My Machine Stalling?

Your machine froze for four seconds, the mouse stuttered, then everything carried on as if nothing happened. Was it memory? The disk? The CPU? Load average shrugs and says "a number went up". Linux PSI (Pressure Stall Information) answers the real question: how much of the last few seconds did work spend waiting?

This is the evergreen companion to our posts on Steam download freezes and ext4 journal modes. We will read the PSI files, learn the difference between some and full, set a kernel-side trigger in C, and then use vmstat and bpftrace to name the culprit once PSI points at a resource.

Why load average is not enough

Load average counts runnable tasks and, on Linux, tasks in uninterruptible sleep, and blends them into one number. It cannot tell you whether those tasks were waiting for a CPU, for a page to come back from swap, or for a disk. It also says nothing about how much useful work was lost.

PSI measures lost time instead. The kernel tracks, per resource, how long tasks were stalled because the resource was contended. Stall time is the quantity you actually feel as lag.

Reading /proc/pressure

On a kernel with PSI enabled you get three files: /proc/pressure/cpu, /proc/pressure/memory and /proc/pressure/io. Each has a some line and (for memory and I/O) a full line. Here is a real read from the sandbox this post was written in:

$ cat /proc/pressure/io
some avg10=0.08 avg60=0.17 avg300=0.05 total=216238
full avg10=0.08 avg60=0.17 avg300=0.05 total=203188

• some: at least one task was stalled on the resource. Capacity is wasted for those tasks, but others kept running.
• full: all non-idle tasks were stalled at once, so the CPUs went unproductive. This is the freeze you notice. For CPU, the system-wide full line is always zero for backward compatibility.
• avg10, avg60, avg300: the percentage of wall time spent stalled over the last 10, 60 and 300 seconds.
• total: cumulative stall time in microseconds. Sample it twice and subtract if you want exact numbers rather than smoothed averages.

Telling the three resources apart

Once a freeze happens, watch all three files together. A tiny loop is enough:

watch -n1 'grep -H . /proc/pressure/*'

• memory some/full spikes: tasks are waiting on reclaim, swap-in or refaulting file pages. Look at direct reclaim and major faults next.
• io some/full spikes with memory quiet: a device is slow or saturated, or writeback is throttling writers. Check iostat -x.
• cpu some high, io and memory quiet: plain CPU contention. Too many runnable threads per core.
• memory and io spiking together: classic thrashing. The page cache or working set no longer fits, so reads hit the disk, which stalls tasks, which stalls memory.

Triggers: let the kernel wake you up

Polling files every second is fine for debugging, but PSI also has an event interface. You open a pressure file, write a threshold, and wait with poll(), select() or epoll(). The format is the type (some or full), the stall amount in microseconds and the window in microseconds. For example, 150000 stall within 1000000 means 150 ms of stall in a 1 second window.

#include <errno.h>
#include <fcntl.h>
#include <poll.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

int main(void) {
    const char trig[] = "some 300000 2000000"; /* 300 ms stalled in 2 s */
    int fd = open("/proc/pressure/memory", O_RDWR | O_NONBLOCK);
    if (fd < 0) { perror("open"); return 1; }
    if (write(fd, trig, strlen(trig) + 1) < 0) { perror("write"); return 1; }

    struct pollfd fds = { .fd = fd, .events = POLLPRI };
    for (;;) {
        if (poll(&fds, 1, -1) < 0) { perror("poll"); return 1; }
        if (fds.revents & POLLERR) { fprintf(stderr, "event source gone\n"); return 1; }
        if (fds.revents & POLLPRI) puts("memory pressure threshold crossed");
    }
}

Each trigger needs its own file descriptor, writing a second trigger to the same descriptor fails with EBUSY, and notifications are rate-limited to one per tracking window. Closing the descriptor removes the trigger. Window sizes run from 500 ms to 10 s.

Per-cgroup pressure: blame the right service

With cgroup v2 mounted, every cgroup directory has cpu.pressure, memory.pressure and io.pressure in the same format. That turns "the server is slow" into "the build container is thrashing". Under systemd the unit cgroup lives below /sys/fs/cgroup:

# which slice is hurting?
for f in /sys/fs/cgroup/*.slice/memory.pressure; do
  printf '%s  ' "$f"; head -1 "$f"
done

# one service
cat /sys/fs/cgroup/system.slice/nginx.service/memory.pressure

Naming the culprit once PSI points at memory

PSI tells you which resource. Counters and tracing tell you why. For memory, start with the cumulative vmstat counters: allocstall counts allocations that had to enter direct reclaim, workingset_refault counts pages that were evicted and then needed again, and pgmajfault counts faults that went to disk.

# snapshot twice, 10 s apart, and diff
grep -E 'allocstall|workingset_refault|pgmajfault' /proc/vmstat > /tmp/a
sleep 10
grep -E 'allocstall|workingset_refault|pgmajfault' /proc/vmstat > /tmp/b
paste /tmp/a /tmp/b | awk '{ printf "%-32s +%d\n", $1, $4 - $2 }'

If allocstall grows, processes are doing reclaim themselves instead of letting kswapd do it in the background. Find out who with a bpftrace one-liner on the direct reclaim tracepoint (run as root):

bpftrace -e 'tracepoint:vmscan:mm_vmscan_direct_reclaim_begin { @[comm] = count(); } interval:s:10 { exit(); }'

The output is a histogram of process names that entered direct reclaim during ten seconds. Usually one or two names dominate: a browser, a game, a build. That is your answer, and the fix is a limit (memory.high on its cgroup), more RAM, or a smaller working set.

Reproduce a stall on purpose

The best way to learn what pressure looks like is to cause some. Run a memory hog inside a throttled cgroup so you do not take down your own session. systemd-run can set MemoryHigh on a transient scope, which makes the kernel reclaim aggressively once the group crosses the line instead of killing it. Use a throwaway machine or VM if you can, and keep the numbers small.

# terminal 1
watch -n1 'cat /proc/pressure/memory'

# terminal 2
systemd-run --user --scope -p MemoryHigh=300M -p MemoryMax=450M   python3 -c "
import time
b = bytearray(600 * 1024 * 1024)
while True:
    for i in range(0, len(b), 4096):
        b[i] = 1
    time.sleep(0.1)
"

Expect the some line to climb while the hog is throttled, and expect the scope itself to show it in its own memory.pressure file. If swap is enabled the hog is pushed into swap and I/O pressure rises with it; if MemoryMax is hit and nothing can be reclaimed, the kernel OOM killer steps in for that cgroup only. That is the lesson in miniature: pressure rises first, and the kill comes later or never.

Now compare the same exercise with the vmstat snapshot above running alongside. You should see allocstall and pgmajfault counters move in step with the pressure numbers, which is how you convince yourself that the metric and the mechanism agree before you trust either in production.

Gotchas

• PSI needs a kernel built with it. If /proc/pressure is missing, your kernel config or boot options disabled it.
• Averages are exponentially smoothed. A one-second hiccup barely moves avg10, so trust the total counter or a trigger for short stalls.
• Pressure is not utilisation. A box can be at 95 percent CPU with zero CPU pressure if nothing is waiting.
• full for CPU at system level is always zero. Do not treat that as good news.

Takeaway

Stop guessing from load average. Read /proc/pressure during the freeze, split the stall into CPU, memory and I/O, then reach for vmstat counters and a bpftrace one-liner to name the process. Add a PSI trigger to your monitoring and the kernel will tell you before your users do.