Safe and Effective Fine-grained TCP Retransmissions for Datacenter Communication

Summary

This paper describes the "TCP incast" problem and proposes several possible solutions.

Incast occurs when a program issues "barrier-synchronized requests" to multiple servers. For example, parallel reads in a distributed FS. The servers all respond at roughly the same time so a large volume of traffic is sent. This causes congestion and large packet loss. Next, TCP goes into retransmission and waits for the RTO of approximately 200ms. Due to this slow retransmission, the link gets very low utilization and the clients cannot progress.

This problem may repeat due to syntonization of the RTO timers, so multiple servers can attempt to retransmit at the same time again causing packet loss and slow retransmission. This issues is particularly acute in data centers because normal transmission latencies are very low, much less than the default RTO of 200ms.

To solve this problem the authors propose to reduce the RTO timer and make it fine grained. The reduced RTO value will increase throughput by waiting for less time after packet loss. However, the authors find that this does not completely solve the problem for large (>1024) sets of servers. The issue is that the timers become synchronized. To avoid this the authors propose randomizing the RTO value.

Thoughts

This paper provides an interesting look at an unexpected problem. I am surprised that the RTO value does not automatically adjust below 200ms based on round trip latency.