End-to-End Arguments in System Design

Summary

This paper argues that providing extra functionality or strong guarantees in the communication system is redundant because applications must still re-implement this functionality at higher levels.

The authors provide a motivating example of a simple file transfer application. The application may fail for a number of reasons including disk errors, crashes, and errors in delivery. The argument is that because of this large set of errors, the application must implement a check for accurate delivery whether or not the communication system does a similar check at a lower level, thus, the low-level check is redundant.

A similar argument are made for secure data transmission. The network is not the only place where data may be leaked, thus encryption must be done at a higher level to protect against all leaks at once. Similarly, suppressing duplicate packets is unnecessary because the application may itself erroneously issue duplicate requests as part of failure/retry procedures.

Thoughts

This paper gets at the key question of how to design abstractions. On one hand, the application has the big picture knowledge of what is happening and what the goals are. In the extreme, this could be used to justify putting all functionality into the application, as in fact it has in the OS world in papers such as Exokernel. On the other hand, the whole point of an abstraction is to separate concerns thus making each piece easier to develop and maintain. This is why modern systems provide basic functionality, such as memory management or scheduling in operating systems, even when it may be more efficient to implement at a higher level.

This paper seems to argue for the "end-point" or application implementing all functionality extreme, however, a careful balance is needed. The paper alludes to this in the performance section when they admit that some reliability at the low-level is necessary to obtain acceptable performance (i.e. don't retransmit a whole file for one lost packet).

Overall, I agree with most of the examples in this paper, however, I believe that a careful balance must be struck between low and high levels to maintain ease-of-use and performance not just eliminating redundancy.