Skip to content

hop 22 of 32 · 5 min

Not overwhelming the receiver

A fast sender and a slow receiver is a problem the network cannot solve. Flow control is the receiver telling the sender how much it can take.

Stop and wait is correct and slow

Send one segment, wait for the acknowledgement, send the next. Correct, and it wastes the link: on a connection with 100ms round trip and 100 Mbps capacity, sending 1,500 bytes and waiting 100ms achieves about 120 kbps. Roughly a thousandth of the link.

Sliding window

Allow several segments in flight at once. The receiver advertises a window: how many bytes it can accept beyond what it has acknowledged.

code
  sent & acked | sent, not acked | may send | cannot yet
  -------------|=================|==========|-----------
               ^                            ^
            window start                window end

As acknowledgements arrive the window slides forward and more may be sent. The window size is carried in every segment, so it adjusts continuously as the application drains the buffer faster or slower.

The number that decides throughput

code
maximum throughput = window size / round trip time

This is the bandwidth delay product, and it explains a class of problem people usually misdiagnose as a slow network.

A 1 Gbps link with 100ms round trip needs a window of 12.5 MB to be filled. The original TCP window field is 16 bits, so it maxes at 65,535 bytes, which on that link gives about 5 Mbps regardless of the capacity available.

Window scaling multiplies the field by a negotiated power of two and removes the ceiling. It is negotiated in the handshake, and a middlebox that strips the option silently caps a fast long link at a few megabits, which is maddening to diagnose because everything reports as working.

Round tripWindow 64 KBWindow 1 MB
1 ms500 Mbpsmore than the link
50 ms10 Mbps160 Mbps
200 ms2.5 Mbps40 Mbps

Silly window syndrome

A receiver whose application reads one byte at a time advertises a one byte window, so the sender sends 1 byte in a 40 byte header. The link fills with overhead.

Both ends avoid it. The receiver does not advertise a tiny window, waiting until a worthwhile amount is free. The sender uses Nagle's algorithm, which holds small writes until the outstanding data is acknowledged or a full segment accumulates.

Nagle's algorithm is also worth disabling deliberately for interactive traffic, with TCP_NODELAY. A game or a terminal wants the keystroke sent now, not batched, and this is one of the few options worth setting by hand.