Fix Bufferbloat on Linux with fq_codel and CAKE

Your internet is fast until somebody uploads a video, and then every call, game and SSH session on the network turns to treacle. Nothing is broken and the link is not saturated in any sensible way. You are watching bufferbloat: queues that are far too deep, quietly turning throughput into latency. This guide reproduces the problem with iperf3 and ping, then fixes it on Linux with fq_codel and CAKE.

What bufferbloat actually is

A router or NIC queue exists to absorb short bursts. Problems start when the queue is huge and the bottleneck link is slow. A bulk TCP transfer keeps the queue full, and every other packet, including your tiny DNS lookup or game update, waits behind it. Throughput looks perfect while latency under load balloons from a few milliseconds to hundreds.

The cure is active queue management (AQM): drop or mark packets early so senders slow down before the queue grows deep, and schedule flows fairly so a bulk transfer cannot starve a small one. RFC 8290 describes FQ-CoDel as a hybrid packet scheduler and AQM algorithm. It hashes packets into per-flow queues, serves them with a modified Deficit Round Robin scheduler and runs the CoDel AQM on each queue.

Reproduce it: measure latency under load

Do not tune what you have not measured. You need two machines, or one machine and a server on the far side of the bottleneck link. Run an iperf3 server somewhere beyond your router, then watch ping while you saturate the link.

# On the far end (a VPS, or a box on the other side of the bottleneck)
iperf3 -s

# On your machine: idle baseline first
ping -c 20 1.1.1.1 | tail -2

# Saturate the UPLOAD direction for 30s and ping at the same time
iperf3 -c YOUR_SERVER -t 30 &
ping -c 25 1.1.1.1 | tail -2
wait

# Now the DOWNLOAD direction (-R reverses the test)
iperf3 -c YOUR_SERVER -t 30 -R &
ping -c 25 1.1.1.1 | tail -2
wait

Compare the max and average round-trip times with the idle baseline. If ping goes from 12 ms idle to several hundred ms under load, you have bufferbloat, and the difference is the number to beat. Write it down. Upload and download behave differently, so test both.

Fix 1: fq_codel on a host

On a single Linux box where the bottleneck is the NIC or a link you do not control, replacing the root qdisc with fq_codel is a one-liner. The tc-fq_codel man page documents these defaults: a 10240 packet hard limit, 1024 flows, a 5 ms target, and a quantum of 1514 bytes. ECN marking is on by default, unlike plain codel, and the interval defaults to 100 ms.

# Replace the root qdisc on eth0 with fq_codel (defaults are sane)
sudo tc qdisc replace dev eth0 root fq_codel

# Inspect it: drops, ecn_mark, new_flows, old_flows
tc -s qdisc show dev eth0

# Make it the default for new interfaces (not persistent across reboot)
sudo sysctl -w net.core.default_qdisc=fq_codel

There is a catch. fq_codel only controls a queue that actually forms in your machine. If the real bottleneck is your ISP modem or DSL line, the queue builds there, out of your reach, and fq_codel on your NIC sees an empty queue because the NIC is faster than the line. To fix that you must create the bottleneck yourself by shaping.

Fix 2: CAKE as a shaper on a router

CAKE (Common Applications Kept Enhanced) bundles a shaper, flow isolation and AQM into one qdisc. Its man page shows the tc-cake options, and several matter for routers: a tiering mode (besteffort, diffserv8, diffserv4 or diffserv3, where diffserv3 is the default), nat or nonat (default nonat), wash or nowash (default nowash), and ack-filter, ack-filter-aggressive or no-ack-filter (default no-ack-filter).

The recipe is to set bandwidth slightly below your real line rate so the queue forms in CAKE instead of the modem. For example, on a 100 Mbit/s down and 20 Mbit/s up connection, shape to roughly 90 percent of what you measure. Start there and tune upward until latency under load starts to climb, then back off a little.

# Upload: shape egress on the WAN interface
tc qdisc replace dev eth0 root cake bandwidth 18Mbit nat

# Check statistics, including per-tin delay and drops
tc -s qdisc show dev eth0

Download is harder because packets have already crossed the bottleneck before your router sees them. The usual trick is to redirect ingress traffic to an IFB (Intermediate Functional Block) virtual device and shape it there with CAKE, again set a little under the line rate.

sudo modprobe ifb numifbs=1
sudo ip link add name ifb4eth0 type ifb 2>/dev/null || true
sudo ip link set ifb4eth0 up

# Send incoming packets on eth0 through ifb4eth0
sudo tc qdisc add dev eth0 handle ffff: ingress
sudo tc filter add dev eth0 parent ffff: matchall \
  action mirred egress redirect dev ifb4eth0

# Shape download there; "ingress" tunes CAKE for traffic already received
sudo tc qdisc replace dev ifb4eth0 root cake bandwidth 90Mbit nat ingress

Verify the fix

Repeat the exact same iperf3 and ping test and compare it with your baseline. A healthy result is that throughput drops only a few percent while latency under load stays close to idle. Then read the qdisc statistics: drops and ECN marks showing up in tc -s output are CoDel doing its job, not a fault.

Gotchas

• Virtual machines and cloud NICs often have their own queueing that you cannot see; test from outside the host.
• Hardware offloads (GSO/GRO) mean the kernel sees large super-packets, which can make per-packet numbers look odd. CAKE splits them by default so fairness stays accurate.
• tc settings do not survive a reboot. Put them in a systemd unit, NetworkManager dispatcher script or your distro's network config.
• Wi-Fi has its own queueing layers, so test on wired first to isolate the problem.
• Running a shaper on a CPU-starved router can itself add latency; watch CPU use at full line rate.

Takeaway

Measure idle latency, measure latency under load, and treat the gap as your score. Use fq_codel where the bottleneck is your own NIC, and CAKE with a bandwidth set slightly below the real line rate where the bottleneck is upstream. Re-measure after every change. It pairs well with reading PSI for memory and CPU stalls: both are about finding where waiting actually happens.