X-Trace: A Pervasive Network Tracing Framework

X-Trace is a debugging tool for tracing a packet through the network across all layers.

Summary

X-Trace is motivated by the need to understand the movement of packets through the complex set of components that makes up a network. It provides a generic mechanism to trace packet movement at all layers of the network stack.

X-Trace works by piggybacking tracing data on top of data packets. Specifically, packet headers are annotated with a task identifier and tree information. The task identifier uniquely identifies the high-level operation being performed. This can tie together many seemingly independent network requests.

The tree information metadata records causal relationships between operations. The network stack is modified to add "pushDown" and "pushNext" function calls. A push down copies metadata to a lower layer, thus ensuring that the metadata stays on the data path. For example, an HTTP proxy calls push down to copy metadata to each TCP connection. Push next is used to propagate metadata to the next hop in the same layer. For example, an HTTP proxy calls push next for each HTTP request it processes. Using these annotations X-Trace is able to construct a tree of operations.

Thoughts

X-Trace is clearly a very useful tool. My main concern is that it requires modifications to protocol implementations, especially to do causal tracking. As the authors explain, the difficulty of this varies a lot from application to application. I wonder if there is a way for the operating system to automate this through "taint tracking."