One Core at 100%? Tune NIC RSS, RPS, RFS, XPS

You bought a 32-core server, put a 25 GbE card in it, and top shows one lonely core at 100% in softirq while the other 31 nap. Welcome to the classic NIC multiqueue problem: the network stack can scale across every core you own, but only if packets are actually spread across queues, interrupts and CPUs. This guide covers Linux NIC multiqueue tuning with RSS, RPS, RFS, XPS and IRQ affinity, so your proxy, WireGuard endpoint or inference server stops bottlenecking on one core.

Why one core melts

A packet arrives, the NIC raises an interrupt on some CPU, and the kernel finishes the heavy lifting in softirq context on that same CPU. If every flow lands on the same receive queue, every interrupt hits the same core, and that core becomes your throughput ceiling no matter how many others are free.

The kernel's networking scaling document lists five techniques that attack this from different angles. Its stated goal is that they should "increase performance uniformly" across CPUs. Here is the map before we touch anything:

• RSS (Receive Side Scaling): the NIC hashes each flow and sends it to one of several hardware receive queues, each with its own IRQ. Hardware-based, so it scales best.
• RPS (Receive Packet Steering): software version of RSS. Useful when the NIC has no RSS or fewer queues than you have CPUs.
• RFS (Receive Flow Steering): steers packets toward the CPU where the consuming application thread runs, for cache locality.
• XPS (Transmit Packet Steering): picks the transmit queue based on the sending CPU, cutting lock contention and cache misses on transmit completion.
• IRQ affinity: which CPUs are allowed to service each queue's interrupt.

Step 1: see what you have

Start by looking at the queues, their interrupts and how softirq work is distributed. Replace eth0 with your interface name.

# How many hardware queues (channels) does the NIC have, and how many are in use?
ethtool -l eth0

# Which IRQ belongs to which queue?
grep eth0 /proc/interrupts

# Which CPUs are actually doing NET_RX / NET_TX work?
watch -d -n1 'grep -E "CPU|NET_RX|NET_TX" /proc/softirqs'

# Per-CPU view of softirq time (mpstat is in the sysstat package)
mpstat -P ALL 1

The kernel documentation says the queue-to-IRQ mapping is visible in /proc/interrupts. If only one eth0 line has climbing counters, or one CPU column in NET_RX grows while the rest stay flat, your traffic is not being spread.

Step 2: RSS, the hardware fix

RSS should be your first stop. The scaling document says that for low-latency networking "the optimal setting is to allocate as many queues as there are CPUs in the system", and with hyperthreading you should limit queues to the number of physical cores. It also notes that NICs should provide an indirection table at least four times larger than the queue count, which is what lets flows be balanced reasonably evenly.

# Use 16 combined RX/TX channels (check the maximum shown by ethtool -l first)
sudo ethtool -L eth0 combined 16

# Show the hash key and the indirection table that maps hash buckets to queues
ethtool -x eth0

# Spread the table evenly across the first 16 queues
sudo ethtool -X eth0 equal 16

Step 3: IRQ affinity and the irqbalance trap

By default an IRQ may be handled by any CPU. You can pin each queue's interrupt with a bitmask in /proc/irq/N/smp_affinity or the friendlier smp_affinity_list. The scaling document warns that irqbalance "may override any manual settings", so decide who is in charge: either let irqbalance manage things, or stop it and pin by hand.

#!/usr/bin/env bash
# Run as root. Assigns eth0 queue IRQs to CPUs 0,1,2,... in order.
set -euo pipefail
systemctl stop irqbalance

cpu=0
for irq in $(grep eth0 /proc/interrupts | cut -d: -f1 | tr -d ' '); do
  echo "$cpu" > "/proc/irq/$irq/smp_affinity_list"
  echo "IRQ $irq -> CPU $cpu"
  cpu=$((cpu + 1))
done

On multi-socket machines, prefer CPUs on the same NUMA node as the NIC (cat /sys/class/net/eth0/device/numa_node tells you which). Cross-node interrupt handling means every packet buffer crosses the interconnect.

Step 4: RPS and RFS, the software fallback

If your NIC has fewer queues than CPUs, or no RSS at all, RPS spreads protocol processing in software by handing packets to other CPUs. It is controlled per queue by a CPU bitmap in rps_cpus, and 0 (the default) disables it. The documentation says that with a multi-queue NIC that already has one queue per CPU, RPS is "probably redundant and unnecessary". On a single-queue device at high interrupt rates, it can be wise to exclude the interrupting CPU from the map.

RFS goes one step further: it records where the consuming application last ran and steers that flow's packets there, improving cache hits. It needs a global socket flow table plus a per-queue count. The kernel documentation says a value of 65536 for rps_sock_flow_entries "works fairly well on a moderately loaded server", and that big servers might need 1048576 or more. With multiple queues, set rps_flow_cnt to rps_sock_flow_entries divided by the number of queues.

# Global RFS flow table (65536 is the documented starting point)
sudo sysctl -w net.core.rps_sock_flow_entries=65536

# Per receive queue: allow CPUs 0-15 (hex mask ffff) and set 65536/16 = 4096 flow entries
for q in /sys/class/net/eth0/queues/rx-*; do
  echo ffff | sudo tee "$q/rps_cpus" > /dev/null
  echo 4096 | sudo tee "$q/rps_flow_cnt" > /dev/null
done

Step 5: XPS for transmit

Transmit has its own contention: many CPUs fighting over the same TX queue lock. The documentation recommends that in a multi-queue system XPS is configured so that each CPU maps onto one queue. If you have fewer queues than CPUs, share queues between CPUs that share a cache.

for n in $(seq 0 15); do
  # hex bitmask with only bit n set
  mask=$(printf "%x" $((1 << n)))
  echo "$mask" | sudo tee "/sys/class/net/eth0/queues/tx-$n/xps_cpus" > /dev/null
done

Step 6: measure, then persist

Re-run the load test with many parallel flows and watch the same counters. You want NET_RX growth spread across the CPUs you chose, no single core saturated in %soft, and throughput or tail latency actually improved. If nothing changed, revert: tuning without a measurable win is superstition.

Gotchas and takeaway

• Do not stack techniques blindly. With one RSS queue per CPU, RPS adds hand-off cost for little gain.
• irqbalance and manual pinning fight. Pick one.
• VMs and cloud NICs may expose fewer queues than the instance has vCPUs; there RPS and XPS are your friends.
• Re-measure after kernel or driver upgrades, since defaults change.

The recipe is simple: use RSS to spread flows across queues, pin or balance those queues' IRQs sensibly, fall back to RPS and RFS when hardware queues run out, map XPS so transmit stays local, and verify with /proc/softirqs. One core at 100% is almost never about the CPU. It is about where the packets went.