Reading TCP congestion control from the kernel

Most explanations of TCP congestion control start with the AIMD diagram and stop there. That is enough to pass an interview, but it does not explain why a link that measures 200 Mbit/s on a speed test stalls at 30 Mbit/s when you pull a single large file. To understand that, you have to look at what the kernel actually does with the congestion window.

The four values that matter

On Linux, three sysctl values and one module decide most of your throughput:

net.ipv4.tcp_congestion_control
net.core.default_qdisc
net.ipv4.tcp_rmem
net.ipv4.tcp_wmem

The first two pick the algorithm and the queue discipline in front of it. The last two set the receive and send buffer ranges, and they are the ones people get wrong.

Why rmem_max overrides tcp_rmem

This is the part that surprises people. tcp_rmem is a three-tuple of minimum, default, and maximum autotuning values. But the autotuner will never grow a buffer past net.core.rmem_max. So if you set:

net.ipv4.tcp_rmem = 4096 131072 6291456
net.core.rmem_max = 212992

the effective ceiling is 208 KB, not 6 MB. The larger number in tcp_rmem is decoration.

Work out what that ceiling costs you. The bandwidth-delay product tells you how much data has to be in flight to keep a pipe full:

BDP = bandwidth x RTT

At 50 ms RTT and a 208 KB window, the best a single stream can do is:

208 KB / 0.05 s = 4.16 MB/s = 33 Mbit/s

That number should look familiar. It is roughly where single-stream transfers plateau on a lot of VPS links, and it has nothing to do with the provider throttling you.

Fixing it

Raise the ceiling and let the autotuner work:

net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 1048576 33554432
net.ipv4.tcp_wmem = 4096 1048576 33554432

Two details worth noting. First, the middle value matters more than people expect: it is what the connection starts with, and a larger starting value shortens the slow-start ramp. Second, 64 MB of maximum is not 64 MB of actual memory — buffers only grow when the window needs them.

Slow start after idle

Long-lived connections that go quiet reset their congestion window, which means every burst after an idle period starts from scratch. For connection-pooled proxies and keepalive HTTP this is pure loss:

net.ipv4.tcp_slow_start_after_idle = 0

Verifying

Do not trust a browser download. Use iperf3 and test both directions separately, because the receive path and the send path are tuned by different sysctls and a one-directional test will hide half the problem:

iperf3 -c host -t 10        # upload
iperf3 -c host -t 10 -R     # download

If upload improves and download does not, you changed tcp_wmem but left rmem_max alone.