Deliverable 3.2:

Abstract

Basic performance measurements on transatlantic high speed lambdas has shown that user level data transfer are hampered by dependencies of the TCP protocol on the properties of the underlying network. Good throughputs can be obtained by making specific choices for Lambda bandwidth provisioning, sophisticated hard- and software, adaptations to the TCP protocol and by the use of multiple data streams. Monitoring tools are selected and presented to show the discrepancy between maximum attainable and application level achieved throughput or goodput.

Executive Summary

The goal of WP 3 task 1 is to understand the dynamics and capabilities of the Lambda based network infrastructure of DataTAG, and provide tools to measure the network capacities and optimize the user's application environment with respect to those networks. This deliverable is a report on the work done under task 1 of workpackage 3. It is the goal of this document to propose a selection of network test and monitor tools that are especially suited for long fat networks. The proposed tools were tested on a Lambda between Amsterdam, located in the Netherlands and Chicago, located in the USA, and later it was tested on the DataTAG circuit between CERN (Switzerland) and Chicago, when it became available. The work was initially focused on the selection and testing of the measurement tools. Since the network has a high bandwidth and a long round trip time, special tools were needed. Recently it has become well understood that many TCP protocol implementations need modifications to cope with long fat network pipes. However, this also means that tools which use TCP to measure performances are as much dependent on the TCP implementation as on the actual properties of the underlying network. Therefore it is very important to precisely define the metrics one wants to measure and subsequently to select the optimal tools for the metric of interest. As we will show later it is for this reason we put emphasis on tools which use UDP for measurements.

The metrics used by the network test tools are defined by the IP Performance Metrics (IPPM) Working Group that is part of the Internet Engineering Task Force (IETF). These documents define specific metrics and procedures for accurately measuring and documenting these metrics. The list of metrics that are of particular interest to the types of networks that we are working with are:

• one-way and round-trip delay and loss
• loss patterns
• bulk transport capacity (achievable throughput) which are especially important for long haul network paths as used in Trans-Atlantic grids.

More information about this work can be found via IP Performance Metrics (IPPM) at the IETF Charter Page.

The network test tools can be divided in two main groups: the network availability test tools and the performance test tools. The availability group is sufficiently supported by (Unix and Windows) system commands such as traceroute and ping. Ping is based on the low level ICMP protocol to determine the round-trip time to a destination (if it is alive), traceroute (Windows "tracrt") determines the Internet path to a destination host. As stated earlier for measuring performances we can choose from two Internet transport protocols those being TCP and UDP. TCP is normally also used by applications to reliably transport data but it suffers from the fact that the measurements is as much dependent on the implementation of the protocol as on the properties of the underlying network. The UDP protocol on the other hand, when used uncontrolled can swamp the network and disturb the normal operation.

The following performance test tools are proposed:

• Netperf (TCP only) to measure bulk data transfer. This tool is easy to use in automated applications due to the server characteristics of the traffic receiver.
• Iperf (TCP and UDP) to measure bulk data transfer. This tool supports many protocols, even multi-cast, and has light weight support of multiple streams.
• UDPmon (UDP only) to measure specific properties of network links, switches, and interfaces. This tool creates a low network load by sending only a fixed timed burst of packets. On the receiver end the time of arrival of every packet is recorded and measurements are based on that information. Network parameters which can be obtained are the peak and average bandwidth, packet loss and detailed packet arival time profiles.

The network availability test tools can be categorized in network equipment and host oriented tools. The network equipment tools use the Simple Network Management Protocol (SNMP) to query all the types of network equipment related to the transfer of the data. Two types of SNMP, network oriented tools are used: Cricket and MRTG. Both tools have recent detailed data information available, while for older data the information level of the data becomes less detailed, dependent on the age of the data.

The host oriented monitor tools selected are SmokePing and rTPL. The SmokePing tool, written by the author of MRTG, can be used to present round-trip time information. It is especially useful to show delay behavior, because it is able to present in one graph all the information related to the RTT distribution. SmokePing is easily extensible by adding external modules. We have added a module to start the RTT commands on remote hosts. This is useful when the host running the SmokePing tool cannot be directly connected to a network to be monitored. The rTPL (remote Throughput Ping Load) package can be used by end users to monitor basic network metrics between a set of workstations. The measured and presented parameters are throughput, round-trip latency, packet loss and host processor load. The results are also presented via the Web, using dynamically generated HTML using a JAVA Applet to read the data from the Web server and present it in graphical form. It should also be noted that historical and average data are available.

The tools described above have been used to test and monitor the single Lambda link between the Amsterdam Science Park (ASP), Amsterdam, the Netherlands and StarLight, Chicago, the USA and when it became available the DataTAG circuit. At first, selected CISCO ONS network equipment for termination of Lambdas as used by Canarie and SURFnet were tested in a back to back fashion. These tests were done in Amsterdam and showed throughputs and performances that matched the provisioned circuits. However, when the equipment was deployed into StarLight and the connection was transatlantic we measured surprisingly low throughputs. In general three types of tests were executed:

• TCP throughput tests to determine the (reliable) bulk transport capacity.
• UDP bandwidth test to obtain an impression of the intrinsic capacity the network.
• Sending a large number of small UDP packets (UDPmon) to obtain detailed latency and loss information. The packet loss results of the UDP tests were verified by using the counters in network equipment, which were queried using the SNMP protocol.

Analysis of this behaviour using the above mentioned tools, in particular UDPmon, showed that the architecture of the ONS equipment played a big role in the observed results. This equipment is capable of accepting Gigabit ethernet and forwarding it on lower rate STM connections and as such it rate limited the links and effectively created bottlenecks. For potentially bursty protocols as TCP this means that massive packet loss due to burst tail drop can occur if buffer space is limited at the bottleneck and packets come in at Gigabit line rates. The data measured by UDPmon could be used to obtain a realistic estimation of the involved memory in the network bottleneck. This research resulted is some simple first order guidelines on path provisioning when using layer two techniques as transport on transatlantic lines. However, it was also shown that even when the equipment in the links had full gigabit ethernet connectivity provisioned, still wild throughput effects could occur. It turned out that in these conditions also the hardware properties of the receiving interface cards and computer bus systems are influential on the observed throughput, as well as the interaction with the protocol stack implementation. The latter was extensively studied in work package 2. From the tests on the available Lambda links the following most important general conclusions could be drawn:

• The TCP protocol used for reliable data transport is (in its standard implementation) unable to achieve a 'high' throughput relative to the capacity of a 'long fat network'
• The UDP protocol is able to sustain a considerable higher throughput but is only under certain conditions acceptable as transport protocol
• Increasing the number of streams improves overall TCP performance.
• The Operating System of the user system can have a significant influence on performance.
• Care has to be taken in the choice of the network interface card (NIC) in regard to an efficient interrupt handling: interrupt coalescence and interrupt latency.
• Further investigation of the behavior of the TCP protocol in regard to 'long fat networks' is absolutely necessary and already undertaken by Work Group 2 of this project.