TCP BBR vs Cubic: Fix Slow Transfers on Lossy Links

A 5 MB file took over a minute to copy between two of our servers. Both ends were idle, both had gigabit ports, and both could reach Cloudflare and Google with zero packet loss. One sysctl on the sending side brought the same copy down to 4.5 seconds.

On that copy it was roughly a 50x speedup from switching the TCP congestion control algorithm from cubic to BBR, and later 20 MB pulls over the same path held 1.4 to 3.4 MB/s instead of about 20 KB/s. BBR is not a fixed multiplier, though: under heavy sustained loss it still degrades, and in the end the bigger win was routing around the lossy hop altogether.

This article walks through how we found the problem, why cubic falls apart on lossy long-haul links, how to enable BBR properly (even on a transfer that is already running), how to get it on CentOS 7, where the stock kernel simply does not have it, and how to bypass the bad hop with a relay.

The symptom: a fast link that moves 20 KB/s

The path in question was a long-haul one, roughly 175 to 220 ms round trip. CI jobs that copy build artifacts across it suddenly took 15 to 60 minutes instead of seconds. A plain 5 MB pull over SSH crawled at about 21 KB/s. We killed it after 60 seconds with only 1.28 MB transferred.

The odd part: the same path in the opposite direction ran at around 670 KB/s. Same wires, same routers, 30x difference depending on who was sending.

That asymmetry is the key clue, and we will come back to it.

Step 1: Is it the network or the hosts? Use mtr from both ends

Before touching any settings, find out where the packets are lost. Run mtr in report mode from both ends, because the forward and return paths on the internet are often different:

mtr --report --report-cycles 100 --tcp --port 443 other-host.example.com

Then run the same thing from the other server back toward the first one.

In our case, both directions showed 15 to 20% packet loss starting in the middle of the path, inside a transit provider’s backbone (a different carrier in each direction). The first few hops, inside each datacenter, were clean.

To rule out the hosts themselves, ping something neutral and well-connected from each end:

mtr --report --report-cycles 100 1.1.1.1
mtr --report --report-cycles 100 8.8.8.8

Both servers showed 0% loss to both targets. So neither host was broken. The loss lived in transit, inside networks we do not control. We could not fix it there, but as it turned out we could avoid it (see “Route around the lossy hop” below).

Rule of thumb: loss that begins at an intermediate hop and continues all the way to the destination is real. Loss at a single hop that disappears on the next ones is usually just a router de-prioritizing ICMP replies, and you can ignore it.

Step 2: Look at the TCP socket on the sender with ss -tin

mtr tells you the path is lossy. ss tells you what TCP is actually doing about it. While the slow transfer is running, look at the socket on the sending side:

ss -tin dst 203.0.113.10

Here is what the slow sender showed (trimmed):

ESTAB 0 1048576  198.51.100.5:22  203.0.113.10:51234
     cubic wscale:7,7 rto:604 rtt:201.3/12.4 mss:1448 cwnd:4 ssthresh:4
     bytes_sent:84213760 bytes_retrans:13416448 delivery_rate:160kbps

What matters here:

• cubic: the congestion control algorithm in use for this socket.
• cwnd:4: the congestion window is 4 segments. At an MSS of 1448 bytes that is under 6 KB in flight per round trip. With a 200 ms RTT, the theoretical ceiling is about 29 KB/s. The observed ~20 KB/s fits.
• ssthresh:4: the slow start threshold has collapsed to the same tiny value, so cubic will not even try to ramp up quickly.
• bytes_retrans:13416448 out of 84 MB sent: about 16% of everything was retransmitted, which matches the loss mtr reported.
• delivery_rate:160kbps: TCP’s own measurement of what it is actually delivering.

The link could carry far more. TCP on the sender had just talked itself into sending almost nothing.

Why cubic collapses on lossy links

Cubic, the Linux default since 2006, is a loss-based algorithm. It treats every lost packet as a sign of congestion and responds by cutting its congestion window. On a clean network that is the right call: loss usually does mean a queue somewhere overflowed.

On a path with steady random loss from a bad transit link, that assumption is wrong. The loss has nothing to do with how fast you send. But cubic keeps halving its window anyway, a lost packet every few round trips, until the window bottoms out at a handful of segments. Combine that with a 200 ms round trip and throughput drops to tens of kilobytes per second.

The longer the RTT, the worse this gets, because cubic needs many round trips to grow the window back and gets knocked down again before it gets there.

What BBR does differently

BBR (Bottleneck Bandwidth and Round-trip propagation time), developed at Google and merged in Linux 4.9, does not use packet loss as its main congestion signal. Instead it continuously measures two things:

• the bottleneck bandwidth: the highest delivery rate it has recently observed
• the minimum RTT: the round-trip time when queues are empty

It then paces packets out at about the measured bandwidth and keeps roughly one bandwidth-delay product in flight. A random 15% loss does not make it believe the pipe has shrunk, so it keeps sending at the rate the path can actually deliver and simply retransmits what was lost.

That is exactly why the reverse direction in our story was fast: that sender was already running BBR. Same lossy path, very different behavior.

The most important point: congestion control is a sender-side setting

TCP congestion control runs on the host that sends the data. The receiver has no say in how fast the sender ramps up.

This trips up a lot of people. If downloads from server A to server B are slow, switching server B to BBR does nothing. You need to change server A. For a git pull, rsync pull, or HTTP download, the sender is the remote server. For a push or upload, the sender is you.

In our case, the slow direction had cubic on the sending host, the fast direction had BBR on the sending host. That is the whole asymmetry.

Step 3: Enable BBR

First check what your kernel offers:

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control

On most modern distributions (Rocky/Alma/RHEL 8+, Ubuntu 18.04+, Debian 10+) BBR is built as a module, so it may not be listed until loaded:

modprobe tcp_bbr
sysctl net.ipv4.tcp_available_congestion_control

net.ipv4.tcp_available_congestion_control = reno cubic bbr

Switch it on for the running system:

sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbr

In our test, the same 5 MB pull went from “killed after 60 s at 1.28 MB” to 4.3 to 4.7 seconds, about 1.1 MB/s. No change on the receiving side, no change to the network. Repeated 20 MB pulls took 5.8 to 14.3 seconds.

Do not expect that to hold for every transfer, though. A long upload over the same path, at about 18% loss, settled at roughly 55 to 250 KB/s on BBR, with the congestion window hovering around 20 to 40 segments. Mainline Linux BBR (v1) still backs off during loss recovery, so heavy sustained loss caps it too. It is a large improvement over cubic’s 20 KB/s, not a cure.

Make it survive a reboot

cat > /etc/sysctl.d/90-bbr.conf <<'EOF'
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF

echo tcp_bbr > /etc/modules-load.d/bbr.conf

sysctl --system

The modules-load.d entry makes sure the module is loaded before sysctl tries to select it at boot.

Two gotchas

Only new connections pick up the change. An already-established socket keeps the algorithm it started with. If a long rsync or download is crawling, changing the sysctl will not rescue it. Restart the transfer if you can. You can confirm with ss -tin: new sockets show bbr where the old ones show cubic.

If you cannot restart it (say, a CI job that would fail and redo an hour of work), you can switch the live socket instead. The congestion algorithm is a per-socket option, TCP_CONGESTION, and any process can set it on its own sockets. With gdb you can make the sending process do exactly that. Find its PID and file descriptor first:

ss -tnp dst 203.0.113.10

ESTAB 0 1749864 198.51.100.5:22 203.0.113.10:60644 users:(("sshd",pid=2730017,fd=4))

Then call setsockopt(fd, IPPROTO_TCP, TCP_CONGESTION, "bbr", 3) inside that process (6 is IPPROTO_TCP, 13 is TCP_CONGESTION):

gdb -p 2730017 -batch -ex 'call (int)setsockopt(4, 6, 13, "bbr", 3)'

A result of $1 = 0 means success, and ss -tin now shows bbr on that socket. On our stuck transfer the congestion window went from 1 to about 800 within seconds and the remaining files finished almost immediately. Treat it as a last resort: gdb pauses the process while attached, and you are poking at a production daemon.

default_qdisc only applies to newly created queues. Your network interface already has a qdisc (often fq_codel), and the sysctl does not replace it. Since Linux 4.13, BBR paces packets on its own and works fine without fq, which is why our test improved immediately. To actually switch the interface to fq without a reboot:

tc qdisc show dev eth0
tc qdisc replace dev eth0 root fq

After a reboot, the sysctl takes care of it.

Scope it to one route instead

If you would rather not change the default for every connection, iproute2 can set the algorithm per route:

ip route replace 203.0.113.0/24 via 198.51.100.1 dev eth0 congctl bbr

Only connections to that network will use BBR. This is handy for a single bad long-haul destination such as an offsite backup target.

CentOS 7: the stock kernel has no BBR

CentOS 7 and RHEL 7 ship kernel 3.10. BBR arrived in 4.9. Red Hat never backported it, so on a stock EL7 box you get:

# modprobe tcp_bbr
modprobe: FATAL: Module tcp_bbr not found.

and tcp_available_congestion_control shows only cubic reno.

That matters more than it sounds. EL7 servers that still run in production often sit in exactly the place where this bites: replication to a distant DR site, backup pushes to another continent, serving downloads to users far away. Every one of those is a sender on a long path, stuck with cubic.

The fix is a newer kernel. Since ELRepo stopped maintaining its EL7 kernels, our Everpatch Kernel provides a maintained kernel-lt for CentOS 7 that tracks the current kernel.org LTS branch. It ships tcp_bbr and the fq qdisc as modules. Follow the install steps on that page, reboot into the new kernel, and confirm:

uname -r
modprobe tcp_bbr
sysctl net.ipv4.tcp_available_congestion_control

Then apply the same /etc/sysctl.d/90-bbr.conf and /etc/modules-load.d/bbr.conf from above. Like most distribution kernels, kernel-lt still defaults to cubic, so this step is required.

You also get a kernel that keeps receiving security fixes, which the 3.10 kernel does not.

Route around the lossy hop

BBR makes the sender tolerate loss. It does not remove it. If the bad stretch is a transit provider’s backbone, the cleanest fix is to send the traffic a different way.

Any third server with clean paths to both ends can act as a relay. Test candidates with plain loss checks on each leg:

# from the sender
ping -c 50 -i 0.2 -q relay.example.com
# from the relay
ping -c 50 -i 0.2 -q receiver.example.com

You want 0% loss on both legs and a total round trip not much longer than the direct one. In our case the direct path lost 15 to 20%, while a relay in a third datacenter had 0% loss on both legs and added almost no latency (98 ms + 63 ms against 172 to 220 ms direct).

For SSH-based transfers (scp, rsync -e ssh, git), the relay needs nothing more than SSH access and TCP forwarding:

rsync -a -e 'ssh -J relay.example.com' receiver.example.com:/data/ /data/

To make it permanent for one destination, put it in ~/.ssh/config. This version falls back to a direct connection if the relay itself is unreachable, so the relay never becomes a single point of failure:

Host receiver.example.com
    ProxyCommand sh -c 'ssh -o ConnectTimeout=8 -W %h:%p relay.example.com 2>/dev/null || exec nc %h %p'

The result on our path, same 20 MB pull, three runs each:

The relay is not only faster on average, it is predictable, which is what CI pipelines need. Each leg is a separate TCP connection over a clean path, so neither one has to fight the loss at all. Keep BBR on the senders anyway: it is cheap insurance for the paths you cannot route around.

When BBR is not the answer

BBR is not a free speed button for every server. Be honest about where it helps:

• It shines on long, lossy paths. High RTT plus random loss is where cubic suffers most and BBR gains most. On a clean LAN or a short low-loss path, you will likely see no difference.
• It does not fix the loss. The packets are still being dropped and retransmitted. If you control the path (your own VPN, your own provider), fixing the lossy link is the better long-term answer. If you do not, a relay that avoids the bad hop (see above) usually beats BBR alone.
• Fairness. The original BBR, which is what mainline Linux ships, can take more than its share of bandwidth when competing with cubic flows through a shallow buffer. On a dedicated server pushing to remote clients that rarely matters. On shared links you care about, measure before and after.
• Higher retransmit counts are normal. BBR keeps sending through loss, so its retransmit numbers can look worse than cubic’s while its throughput is far better. Judge it by delivery_rate and real transfer times, not by retransmits alone.

Quick checklist

When a transfer between two otherwise healthy servers is mysteriously slow:

1. Run mtr --report from both ends, plus to 1.1.1.1 from each end, to tell transit loss from host problems.
2. On the sending host, run ss -tin dst <peer> during the transfer. A tiny cwnd, high bytes_retrans, and a low delivery_rate point to loss-based congestion control giving up.
3. Enable BBR + fq on the sender, persist it in /etc/sysctl.d/ and /etc/modules-load.d/.
4. Restart the transfer, since established sockets keep cubic. If you cannot, switch the live socket with gdb and setsockopt(TCP_CONGESTION).
5. On CentOS 7, install the Everpatch Kernel first, because the stock 3.10 kernel cannot do BBR at all.
6. If the loss sits in someone else’s transit, find a relay with clean paths to both ends and route SSH through it with ProxyJump or a ProxyCommand with a direct fallback.

One sysctl on the right host turned a minute-long copy into a few seconds, and a relay made it fast every time. The hard part was knowing which host to look at.