Slow Linux Disk? NVMe, blk-mq and I/O Schedulers

Everyone has a favourite Linux I/O scheduler, and most of those opinions were formed on spinning disks in 2011. If your NVMe drive feels slow, the answer is rarely a magic echo into sysfs, but understanding the Linux I/O scheduler choice (none, mq-deadline or kyber) tells you where requests queue up, and a ten-line fio job tells you whether your change helped or you just fooled yourself.

How blk-mq moves a request

Modern Linux block I/O goes through blk-mq, the multi-queue block layer. The kernel documentation describes it as an API for letting fast storage reach a huge number of IOPS by queueing and submitting requests to the device in parallel. The old design had one queue with one lock, which did not scale on SMP machines; blk-mq instead gives you many queues with CPU-local entry points.

There are two groups of queues. Software staging queues sit near the CPUs and are where requests get merged and, if a scheduler is attached, scheduled. Hardware dispatch queues are the second stage that the device actually pulls from. The documented shortcut matters: a request takes the shortest path straight to the hardware queue unless an I/O scheduler is attached or the layer wants to try merging. That is the whole reason none exists. With no scheduler, there is nothing between your application and the device queues.

Meet the schedulers

The kernel's switching-scheduler document lists the options you can select on a block device on the fly: mq-deadline, none, bfq and kyber. The sysfs file /sys/block/DEV/queue/scheduler lists the valid names and brackets the active one.

• none: no reordering and no extra queueing. Lowest CPU overhead and the common choice for fast NVMe, though you should check what your own kernel picked.
• mq-deadline: gives each request a deadline and prefers reads over writes while still guaranteeing writes are not starved. Sensible for SATA SSDs and HDDs, and for mixed workloads where read latency matters.
• kyber: a small latency-targeting scheduler. Its only two tunables are target latencies for reads and synchronous writes, and it throttles requests to meet them.
• bfq: a fairness-oriented scheduler with per-process weighting, mostly of interest on desktops and slow media. It is out of scope for the benchmarks here.

What the mq-deadline knobs actually do

The deadline scheduler documentation defines its tunables in /sys/block/DEV/queue/iosched. read_expire and write_expire are the deadlines, in milliseconds, assigned when a request enters the scheduler. fifo_batch is the maximum number of requests per same-direction batch; the docs say smaller values favour latency (1 gives first-come first-served behaviour) while larger values improve throughput at the cost of latency variation. writes_starved is how many times reads are preferred before writes get dispatched, and front_merges set to 0 disables the front-merge lookup.

Look before you tune: reading the evidence

Before touching anything, find out whether the disk is the bottleneck. The kernel's iostats documentation defines the counters in /proc/diskstats. Field 9 is the number of I/Os currently in progress, and field 10 is milliseconds spent doing I/Os, which the documentation says keeps increasing as long as field 9 is nonzero. That detail is why the percent-utilisation column in iostat can sit at 100 on an NVMe drive that still has plenty of parallel headroom: it only says the device was never idle, not that it was saturated.

DEV=nvme0n1

cat /sys/block/$DEV/queue/scheduler
cat /sys/block/$DEV/queue/nr_requests
ls /sys/block/$DEV/mq | wc -l        # number of hardware queues
cat /sys/block/$DEV/queue/rotational  # 0 = SSD/NVMe

# live view: watch queue depth and latency, not just %util
iostat -x $DEV 1

In the iostat extended output, read the average queue size and the await columns together. Rising queue size with rising await means requests are waiting in line; a high await with a tiny queue points at the device or the filesystem layer, not the scheduler. Pair this with the off-CPU view from our flame graph post to see whether your application is the one stuck waiting.

Switch at runtime and make it stick

echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler
cat /sys/block/nvme0n1/queue/scheduler
# [mq-deadline] none kyber bfq    (list depends on your kernel)

# deadline tunables
cat /sys/block/nvme0n1/queue/iosched/read_expire
echo 100 | sudo tee /sys/block/nvme0n1/queue/iosched/read_expire

These writes are runtime only and vanish at reboot. To persist a choice, use a udev rule that matches the device kind rather than a name, since NVMe names can change between boots. Treat the rule below as a template and test it with udevadm before relying on it.

# NVMe: no scheduler
ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", ATTR{queue/scheduler}="none"

# Rotational disks: mq-deadline
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline"

sudo udevadm control --reload
sudo udevadm trigger --subsystem-match=block --action=change
cat /sys/block/*/queue/scheduler

Prove it with fio

fio documents direct=bool as non-buffered I/O (usually O_DIRECT, default false) and iodepth as the number of I/O units to keep in flight, noting that depth beyond 1 does not affect synchronous engines. So use an asynchronous engine and bypass the page cache, otherwise you are benchmarking RAM. ramp_time lets performance settle before numbers are logged, and clat_percentiles reports completion latency percentiles, which is what you should compare. Averages hide tail latency, and tails are what users feel.

#!/usr/bin/env bash
set -euo pipefail
DEV=nvme0n1
TEST=/mnt/fiotest/file    # a file on the filesystem you care about

for s in none mq-deadline kyber; do
  echo "$s" | sudo tee /sys/block/$DEV/queue/scheduler >/dev/null
  echo "=== $s ==="
  fio --name=randread --filename=$TEST --size=4G \
      --rw=randread --bs=4k --ioengine=io_uring --direct=1 \
      --iodepth=32 --numjobs=4 --group_reporting \
      --time_based --runtime=30 --ramp_time=5 \
      --clat_percentiles=1 --percentile_list=50:99:99.9 \
      | grep -E "IOPS|99.00th|99.90th"
done

Run it at least three times and alternate the order, because thermal throttling and drive caches will otherwise award a win to whichever scheduler ran first. Then repeat with a mixed workload (rw=randrw, rwmixread=70) since schedulers differ most when reads and writes compete. If the numbers are within run-to-run noise, keep the default. That is a legitimate result, and the cheapest one.

Gotchas

• Check the filesystem and writeback first. Slow writes are often dirty-page throttling, not the block layer; see our dirty-page tuning post.
• Do not test on the root filesystem you depend on, and never point fio at a raw device that holds data when you run write jobs.
• The scheduler list depends on the kernel config. If a name is missing from the sysfs file, the module or option is not available.
• Kyber's latency targets are in nanoseconds. Set them from measured numbers, not from a blog post, including this one.
• Virtual disks (cloud volumes, virtio) have their own queues underneath, so results on a VM may not transfer to bare metal.

Takeaway

Treat the I/O scheduler as one dial among several. Read the queue and latency evidence first, try none on fast NVMe and mq-deadline where reads need protection from writes, reach for kyber only with a latency target you measured, and judge every change by tail latency in a repeatable fio run. If nothing moves outside noise, congratulations: your disk was never the problem, and you saved yourself a pile of cargo-cult sysfs lines.