A short note on socket buffer sizing

Buffer sizing looks like a tuning knob but it is really a latency decision. Every byte sitting in a buffer is a byte that has already been read from the sender and not yet delivered to the receiver. Bigger buffers raise throughput on long fat links and raise queueing delay everywhere else.

The rule

A buffer needs to be at least one bandwidth-delay product to keep a link full:

buffer >= bandwidth x RTT

For a 100 Mbit/s link at 20 ms that is about 250 KB. For the same link at 200 ms — a satellite or a badly routed international path — it is 2.5 MB. Same bandwidth, ten times the buffer requirement, because the pipe is ten times longer.

Where the numbers live

net.core.rmem_max      # hard ceiling for receive buffers
net.core.wmem_max      # hard ceiling for send buffers
net.ipv4.tcp_rmem      # min / default / max for autotuning
net.ipv4.tcp_wmem      # min / default / max for autotuning

The tcp_* tuples are ranges the kernel autotunes within. The core.* values are absolute caps. The effective maximum is the smaller of the two, which is why setting a large tcp_rmem max while leaving rmem_max at its 208 KB default accomplishes nothing.

Bufferbloat is the other direction

Oversized buffers on a bottleneck link turn into seconds of standing queue. The fix is not smaller buffers but a queue discipline that drops or marks early:

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

BBR in particular does not rely on loss as its primary signal, so it behaves well with fq and degrades badly with a deep FIFO. Pairing BBR with a large unmanaged buffer gives you the worst of both.

What to actually check

Watch the socket's own view rather than guessing:

ss -tmn state established '( dport = :443 )'

The skmem field shows allocated versus queued bytes. If the receive queue is consistently pinned at its limit while the send side has headroom, the receive buffer is your bottleneck and raising rmem_max will help. If both are small and idle, buffers were never the problem and you should be looking at loss or the congestion algorithm instead.

Do not set defaults to the maximum

A tempting mistake is setting rmem_default and wmem_default to the ceiling. Every connection then reserves that much on creation, and a box handling a few thousand concurrent sockets runs out of memory for no benefit — most of those connections are short and will never fill the buffer. Set the default to something modest and let autotuning grow it.