io_uring in Practice: When Async I/O Wins

Every read() you have ever written is a tiny negotiation with the kernel: stop what you are doing, cross the user and kernel boundary, wait, come back. io_uring is Linux saying: what if you just left a note on the counter and picked up the results later? This post explains how io_uring works, shows a minimal liburing program, walks through the setup flags that change behaviour, and covers the one sysctl you should know before you ship it, because io_uring is also a big chunk of kernel attack surface.

The problem: syscalls and readiness are not the same as completion

Classic Linux I/O comes in two flavours. Blocking calls (read, write) park your thread until the data is ready. Readiness APIs (select, poll, epoll) tell you a socket is ready, after which you still make a syscall per operation to actually move bytes. That model works well for sockets, but regular files are awkward: they are always reported as ready, so epoll does not help with disk reads, and people ended up with thread pools to fake asynchrony.

io_uring is a completion-based interface. You describe the operation you want (read this file at this offset, accept on this socket), the kernel performs it, and you collect a result. Nothing about it is limited to sockets. The same ring takes file reads, writes, accepts, sends and more, and you can queue many operations and submit them in one syscall.

How the rings work

An io_uring instance is a pair of ring buffers shared between your process and the kernel. You put submission queue entries (SQEs) on the submission queue (SQ), and the kernel puts completion queue entries (CQEs) on the completion queue (CQ). Because the rings live in shared memory, filling them needs no syscall; you only enter the kernel to tell it new work is waiting (io_uring_enter), to wait for completions, or both at once.

• io_uring_setup creates the instance and returns a file descriptor; the rings are mapped into your address space.
• An SQE describes one operation: an opcode, a file descriptor, a buffer, an offset and a user_data value that you pick.
• A CQE carries that same user_data back plus the result, which is the syscall-style return value or a negative errno.
• Completions can arrive in any order, so user_data is how you match results to requests.

Hands-on with liburing

You can drive the rings with raw syscalls and mmap, but almost everyone uses liburing, the helper library maintained alongside the kernel interface. Its README says it sets up and tears down instances and offers a simplified interface, and that it is not tied to a kernel release, so newer liburing runs on older kernels, though newer features need newer kernels. It builds with the usual configure, make and make install sequence, and most distributions package it as liburing-dev or liburing-devel.

The call sequence below follows the one used by the io_uring-cp example in the liburing repository: initialise a ring, get an SQE, prepare the operation, submit, wait for a CQE, mark it seen, and tear the ring down.

#include <liburing.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char **argv)
{
    if (argc < 2) {
        fprintf(stderr, "usage: %s FILE\n", argv[0]);
        return 1;
    }

    int fd = open(argv[1], O_RDONLY);
    if (fd < 0) {
        perror("open");
        return 1;
    }

    struct io_uring ring;
    int ret = io_uring_queue_init(8, &ring, 0);
    if (ret < 0) {
        fprintf(stderr, "queue_init: %s\n", strerror(-ret));
        return 1;
    }

    static char buf[4096];
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_read(sqe, fd, buf, sizeof(buf), 0);
    io_uring_sqe_set_data(sqe, buf);

    ret = io_uring_submit(&ring);
    if (ret < 0) {
        fprintf(stderr, "submit: %s\n", strerror(-ret));
        return 1;
    }

    struct io_uring_cqe *cqe;
    ret = io_uring_wait_cqe(&ring, &cqe);
    if (ret < 0) {
        fprintf(stderr, "wait_cqe: %s\n", strerror(-ret));
        return 1;
    }

    if (cqe->res < 0)
        fprintf(stderr, "read failed: %s\n", strerror(-cqe->res));
    else
        printf("read %d bytes\n", cqe->res);

    io_uring_cqe_seen(&ring, cqe);
    io_uring_queue_exit(&ring);
    close(fd);
    return 0;
}

gcc -O2 -o read_one read_one.c -luring
./read_one /etc/os-release

One read is not a good demonstration, since a plain pread would be just as fast. The payoff comes from queue depth: call io_uring_get_sqe and the prep function in a loop for 32 reads, make a single io_uring_submit call, and then reap completions as they arrive. That is one syscall to start 32 operations, and the io_uring-cp example batches its reads the same way before submitting once.

Benchmark it before you believe it

Whether io_uring wins depends on your workload, so measure instead of trusting blog posts, including this one. fio has an io_uring engine, which makes an honest comparison easy: same file, same block size, same queue depth, different engine. Use direct I/O so the page cache does not flatter either result.

for engine in psync libaio io_uring; do
  echo "== $engine =="
  fio --name=randread --filename=./fio.test --size=2G \
      --rw=randread --bs=4k --direct=1 \
      --ioengine=$engine --iodepth=32 \
      --runtime=30 --time_based --group_reporting
done

• Expect little difference at queue depth 1; the benefit of batching needs depth.
• Fast NVMe devices and high IOPS are where per-operation syscall overhead is most visible.
• For network servers, compare against your real epoll loop with realistic connection counts, not a loopback toy.
• Run each test several times and watch CPU use as well as throughput; fewer syscalls can mean the same throughput for less CPU.

Setup flags that change behaviour

The flags you pass at setup change how work is scheduled. The liburing man page for io_uring_setup documents these, with kernel versions where it states them.

• IORING_SETUP_SQPOLL: a kernel thread polls the submission queue so you can submit without syscalls. The thread goes to sleep after sq_thread_idle milliseconds and you must then call io_uring_enter to wake it. Per the man page, kernel 5.11 dropped the fixed-file requirement and 5.13 removed the need for special privileges.
• IORING_SETUP_SINGLE_ISSUER (kernel 6.0): a hint that only one task submits requests, which lets the kernel optimise. Requests fail with -EEXIST if you break the rule.
• IORING_SETUP_COOP_TASKRUN (kernel 5.19): the kernel stops interrupting your userspace task when a completion arrives; completions are processed at the next kernel and user transition instead.
• IORING_SETUP_DEFER_TASKRUN (kernel 6.1): completion work is deferred until io_uring_enter is called with IORING_ENTER_GETEVENTS. It requires SINGLE_ISSUER, and the submitting thread must make the call.

Gotchas and the security knob

io_uring exposes a very large amount of kernel functionality through one interface, and that is why some environments restrict it. The kernel provides a sysctl for this, documented in the kernel admin guide: io_uring_disabled prevents processes from creating new io_uring instances, and the documentation says enabling it shrinks the kernel attack surface. Existing instances keep working when it is changed.

• 0: all processes can create io_uring instances. This is the default.
• 1: creation is disabled for unprivileged processes that are not in the io_uring_group group.
• 2: creation is disabled for all processes.
• io_uring_group: with the value 1, a process needs CAP_SYS_ADMIN or membership in this group; the default of -1 means only CAP_SYS_ADMIN processes may create instances.

# current setting
sysctl kernel.io_uring_disabled

# allow only a dedicated group
sudo groupadd --system uring
sudo sysctl -w kernel.io_uring_group=$(getent group uring | cut -d: -f3)
sudo sysctl -w kernel.io_uring_disabled=1

# persist
printf 'kernel.io_uring_disabled = 1\n' | sudo tee /etc/sysctl.d/60-io-uring.conf

Takeaway

Reach for io_uring when you have many in-flight operations, fast storage, or files and sockets you want under one completion loop. Stay with epoll or plain syscalls when concurrency is low and the code simplicity matters more. Either way, benchmark with fio at your real queue depth, pick the setup flags deliberately, and decide on purpose whether the containers and hosts you run should be allowed to create rings at all.