systemd 262: Tame Restart Storms and Boot Stampedes

Picture fifty services crashing at the same instant, all waiting exactly five seconds, and all restarting at the same instant, straight into the same dead database. That is a restart storm, and systemd 262 finally ships two small knobs aimed at it: RestartRandomizedDelaySec= and ActivatingConcurrencyMax=. This post walks through both with runnable examples, covers the new static PID 1 build for tiny containers, and lists the incompatible changes that can bite you on upgrade.

The problem: synchronized failure is a thundering herd

Restart=on-failure with RestartSec=5 is deterministic. If a shared dependency dies and takes a whole fleet of units with it, they all fail within the same few milliseconds and all retry on the same schedule. Exponential backoff makes the retries slower but keeps them in lockstep, so the herd just stampedes less often. What you actually want is jitter: spread the retries out so the recovering dependency sees a trickle instead of a wall.

The same shape appears at boot. Start a hundred heavy units at once and you get disk thrash, memory spikes and timeouts that are really just contention. systemd has always parallelised aggressively, and until now the only brakes were ordering dependencies and cgroup limits.

RestartRandomizedDelaySec: jitter for restarts

According to the systemd 262 NEWS file, service units gained RestartRandomizedDelaySec=, which adds a uniformly distributed extra delay to automatic restarts on top of RestartSec= and optional exponential backoff. The stated purpose is to spread out synchronized failure and restart storms. So with RestartSec=5 and a randomized delay of 10 seconds, each restart waits somewhere in the 5 to 15 second window rather than exactly 5.

[Unit]
Description=Queue worker %i
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/worker --id %i
Restart=on-failure
RestartSec=5
RestartRandomizedDelaySec=10
# Give up if it keeps failing, so a bad deploy does not loop forever
StartLimitIntervalSec=300
StartLimitBurst=10

[Install]
WantedBy=multi-user.target

You can see the effect without waiting for a real outage. Make a throwaway unit that exits immediately, start twenty instances, and watch the restart timestamps fan out in the journal.

systemctl --version | head -1

for i in $(seq 1 20); do
  sudo systemd-run --unit="jitter-test-$i" \
    -p Restart=always -p RestartSec=2 -p RestartRandomizedDelaySec=8 \
    /bin/false
done

# Each unit restarts at a different moment inside the 2-10 second window
journalctl -u 'jitter-test-*' --since '1 min ago' -o short-precise | grep -i 'scheduled restart'

# Clean up
for i in $(seq 1 20); do sudo systemctl stop "jitter-test-$i"; done

ActivatingConcurrencyMax: a bouncer for slices

The second knob lives on slice units. Per the NEWS file, ActivatingConcurrencyMax= limits how many units in a slice hierarchy may be in the activating state at the same time. Further activations are queued and started automatically as slots free up. Note what it counts: units that are still starting, not units that are running. Once a unit finishes activating, its slot is released.

That makes it a good fit for expensive start-up work such as services that warm a cache, replay a journal, or load a multi-gigabyte model from disk. Put them in one slice and cap how many can be mid-start together.

# heavy-start.slice
[Unit]
Description=Services with expensive start-up

[Slice]
ActivatingConcurrencyMax=2

# [email protected]
[Unit]
Description=Model loader %i

[Service]
Type=notify
Slice=heavy-start.slice
ExecStart=/usr/local/bin/model-loader --model %i
TimeoutStartSec=300

sudo systemctl daemon-reload
sudo systemctl start --no-block model-loader@{a,b,c,d,e,f,g,h}.service
systemctl list-jobs
# Expect: two jobs running, the rest waiting for their turn

Static PID 1 for very small containers

Two related changes make systemd viable in minimal containers. First, the manager now embeds a basic set of unit files (basic.target, sysinit.target, multi-user.target and friends) and falls back to them when nothing can be loaded from disk, with on-disk units and masks always taking precedence. A container can therefore start with systemd as PID 1 without installing any unit files. Second, systemd can be built as a single statically linked PID 1 and executor binary, configured with Meson as documented in the release notes.

meson setup build-static \
  --default-library=static --prefer-static \
  -Dbuild-static=true \
  -Dsystemd-multicall-binary=true
ninja -C build-static

The trade-offs are spelled out too: such builds do not use dlopen() to load optional libraries at runtime, and they use simplified passwd and group lookup instead of NSS. If your workload relies on NSS modules such as LDAP or sssd for user resolution, a static build is not for you.

Upgrade gotchas worth a grep

Every systemd release hides a few changes that only hurt in production. These are the ones from the 262 incompatible-changes list most likely to affect ordinary servers. Each is quoted from the NEWS file in spirit, so read the original before you roll out.

• UnsetEnvironment= is now applied after environment variable expansion of ExecStart= and related lines. A variable you unset can still be used in the command line, but is absent from the process environment. Units that relied on whole-word $VAR references being dropped need updating.
• Services using Type=notify-reload must now catch or block ReloadSignal= when they send READY=1, or the unit fails to start with a protocol error.
• systemd-sysupdate.service and systemd-sysupdate.timer were renamed to systemd-sysupdate-update.service and systemd-sysupdate-update.timer, with compatibility symlinks provided. The systemd-sysupdate binary moved back to libexecdir and is marked experimental again.
• Rate-limit timers such as StartLimitIntervalSec= now use CLOCK_BOOTTIME instead of CLOCK_MONOTONIC, so suspended time counts. State serialized by an older manager may expire once earlier than before after the upgrade.
• journalctl now rejects -F/--field combined with filters such as unit, boot, time, cursor or grep, which were previously silently ignored.
• TPM-sealed credentials are now pinned to the TPM's SRK. Credentials minted after the change are not recognised by older systemd versions, although new systemd still accepts older ones.
• The tpm2-measure-bank= crypttab option is deprecated and has no effect.

# Units that use UnsetEnvironment or notify-reload
sudo grep -rIl --include='*.service' -e 'UnsetEnvironment=' -e 'Type=notify-reload' /etc/systemd /usr/lib/systemd 2>/dev/null

# Scripts that combine journalctl --field with filters
sudo grep -rIn -E 'journalctl .*(-F|--field).*(-u|-b|--since|--grep)' /usr/local /opt /etc 2>/dev/null

# Anything hard-coding the old sysupdate unit names
sudo grep -rIn 'systemd-sysupdate\.\(service\|timer\)' /etc /usr/local 2>/dev/null

Takeaway

• Add RestartRandomizedDelaySec= to fleets of similar services that depend on one shared backend.
• Cap heavy start-up work with a slice and ActivatingConcurrencyMax= instead of hand-written ordering hacks.
• Consider the static PID 1 build only if you can live without dlopen() and NSS.
• Grep for UnsetEnvironment=, Type=notify-reload and old sysupdate unit names before upgrading.
• Check your distribution's systemd version first, since both new settings need 262 or newer.