Tbwtools: Debugging TCP Performance

CESNET technical report number 6/2006
also available in PDF, PostScript, and XML formats.

Sven Ubik, Václav Řehák, Lukáš Baxa
6.10.2006

1   Abstract

In this report we describe tbwtools, an integrated set of tools that we developed to assist in debugging poor performance of TCP data transfers.

Keywords: TCP performance, congestion control, network performance

2   Introduction

Insufficient performance of TCP data transfers can be a result of one or more of the following conditions, going from the network line up to the software application:

In order to detect the bottleneck element, we usually use a combination of tools and techniques, such as the following (not an exhaustive list):

The last four tests are usually done only once to check if that is the problem or not. On the other hand, the analysis of tcpdump tracefile and watching TCP runtime variables often requires multiple iterations (due to variations of concurrent network traffic) and manual correlation of information from several sources, which makes it a time-consuming task.

Therefore, we decided to develop a set of tools and a framework called tbwtools, for integrated debugging of TCP performance using information obtained from several sources. Tbwtools use a combination of active and passive monitoring and are installed entirely on end hosts, no access to network infrastructure (routers, etc.) is required.

The rest of this report is organized as follows. We state requirements to be fulfilled by our framework in Section. We shortly describe certain tools developed by other people that we use in our framework in Section. Then we describe the architecture of tbwtools in Section and the individual tools comprising our framework in Section. We mention some implementation and operation details in Section. Finally, we present an example of use of tbwtools in Section.

3   Requirements

Based on our experience with debugging performance of TCP transfers, we set the following features to be supported by out framework:

We assume the following use cases:

4   Related tools

In this section we shortly describe tools and systems developed by other people that we use in our framework or that we want our framework to be compatible with.

4.1   web100

Web100 is a tool for the Linux operating system that enables access to TCP runtime variables and to a limited extend also configuring some TCP-related kernel parameters. Some of the monitored TCP runtime variables are already present in the standard Linux kernel and some are added by Web100. Web100 consists of three components:

There are several versions of web100 compatible with different Linux kernel versions. If you want to include also PPSkit (for precise time synchronization) or AIMD patch (to configure and monitor TCP congestion control) into the same kernel you must choose a Linux kernel version for which all required patches are available.

As of now, the latest Linux kernel version supported by web100 is 2.6.16, the latest version supported by PPSkit is 2.6.15.1 and the latest version supported by AIMD patch is 2.6.9.

In order to integrate web100 into our framework, we developed a new tool called wread to access TCP runtime variables of a specified connection, which is described later in Section.

4.2   perfSONAR

perfSONAR [perfsonar] system is a set of services to facilitate multi-domain performance monitoring with different measurement tools. perfSONAR does not include measurement tools themselves. It is a layer between measurement tools and the user. Services of perfSONAR care about data storage, authentication, registration and look-up of components distributed across a network. More services are planned to be added in the future, such as a general data transformation service. perfSONAR is being developed by JRA1 activity of the GN2 project and the first release, which includes a subset of services is available for download at the perfSONAR website.

Tbwtools are integrated with perfSONAR through one of its services - the Measurement Point (MP) service, which wraps up different measurement tools. There is one type of MP for each supported measurement tool. We have developed tbwMP measurement point, which communicates with tbwtools on one side and with perfSONAR on the other side. Communication with perfSONAR is done by NMWG-based XML messages [nmwg]. Network Measurement Working Group (NMWG) is a part of Global Grid Forum (GGF). It intends to identify network communication characteristics important for grid applications and specify them with XML schemas suitable for monitoring applications.

Both perfSONAR and tbwtools integrate monitoring tools in some sense. PerfSONAR treats monitoring tools separately and adds higher-level services to them. Tbwtools integrates the functionality of related monitoring tools so that they can be used together as one tool. This integrated tool then communicates with perfSONAR.

5   Architecture

Communication between tbwtools components is illustrated in Figure. User interface is implemented as a Java applet, called tbwApplet, which can be downloaded from a web server and opened in a web browser.

Performance debugging is done in two phases. In the first phase, a user instructs tbwApplet to make a test connection. In the second phase, monitoring data gathered during the test connection is analysed and presented.

Phase 1 - measurement

A test connection can be made a) between user s workstation and some remote PC or b) between two remote PCs. Note that there are arrows on both ends of the lines illustrating test connections in Figure. This is because a test connection can be inititated in either direction, to test performance of the forward or backward path.

A test connection is performed by a tool called bulk. It is a standalone stress-type throughput measurement tool (similar to iperf) with built-in measurement capabilities. TbwApplet also includes a bulk implementation in a separate thread. This allows for making a test connection also to and from the user s PC. Only the part of the bulk tool which makes a test connection is integrated in tbwApplet. The monitoring part is not included, because tbwApplet is expected to be run mostly in the Windows operating system, which does not provide access to socket options needed by bulk.

[Figure]

Figure 1: Tbwtools architecture

To communicate with tbwApplet, the remote PC runs tbwMP measurement point compatible with perfSONAR. Messages exchanged between tbwApplet and tbwMP and related actions initiated by tbwMP in the first phase are illustrated in Figure. Messages are in XML format based on NMWG recommendations, as are all other messages between services in perfSONAR. TbwApplet sends a MetadataKeyRequest message to tbwMP to request a test connection and its monitoring. TbwMP starts tcpdump and wread for monitoring and bulk to make a test connection. A measure script (see Section) is used to start these tools with proper arguments. After the test connection finishes, tbwMP stops tcpdump and wread and sends a MetadataKeyResponse message with data ID back to tbwApplet.

The use of tbwApplet is not necessary. A user can also use perfSONAR client to send and receive XML messages to and from tbwMP. Of course, these messages must be prepared by the user and a test connection can be made between two remote PCs only.

[Figure]

Figure 2: Phase 1 - making a test connection

Phase 2 - data analysis and presentation

Messages exchanged between tbwApplet and tbwMP and related actions in the second phase are illustrated in Figure. The user selects in tbwApplet what characteristics should be extracted from the measurement data and sends the list of these characteristics along with the data ID to tbwMP in a MeasurementRequest message. The measurement data is processed by tbwsyn utility, graphs are plotted by gengraph script and uploaded to the web server. In the current release, the web server is located on the machine where tbwMP is running. In the future release, which is under development, the web server will be separated from tbwMP and can be located anywhere. User can now request results from the web server. Summary information about the test connection is also shown including, for example, the percentage of time when throughput was limited by the sender, the receiver or the network. Examples of XML messages between tbwApplet and tbwMP in the two phases of operation and their formal syntax in RELAX NG are included in Appendix A.

[Figure]

Figure 3: Phase 2: processing and presenting measurement results

6   Tools

In this section we describe the individual tools that comprise the tbwtools framework. A user can use each tool individually. However, a more convenient option is to interact with tbwtools through the user interface provided by the tbwApplet.

6.1   bulk2

We described bulk tool in more detail in [bulk]. It is a stress-type throughput measurement tool, which can run as a server on one machine and as a client on another machine, with built-in support for reporting current values of any socket options as requested by the user. Bulk can also write values to socket options, when write access is supported by the particular options.

Particularly interesting is the TCP_INFO socket option at the SOL_TCP level. Requested socket options and their data fields can be specified by mnemonics available for the existing socket options or by numbers for any possible future socket options.

Bulk also recognizes the new TCP_AIMD and TCP_COUNTERS socket options added by the AIMD patch (which is distributed with bulk in the same package) to configure and monitor TCP congestion control.

Bulk reads requested socket options after each call to send() on the sender or recv() on the receiver. A user can alter frequency of reporting by setting the volume of data, which is to be sent or received in one call by -b command-line argument.

Some information provided by bulk overlaps with web100 TCP runtime variables. An important advantage of bulk is that it can obtain a lot of interesting runtime information about TCP transfers without the need to patch the kernel (as web100 requires) or root privileges.

For example, the following command can be used to initiate a test connection to perfmon1.cesnet.cz, with data sent in 16 kB blocks and request periodic reports of congestion window, rtt, and rto variables: 

$ bulk2 -b16384 -gSnd_Cwnd,rtt,rto perfmon1.cesnet.cz

Note that the socket option name can be omitted here as it is not ambiguous (TCP_INFO).

We added the following functionality to the new version bulk2 for the purposes of integration within the tbwtools framework:

6.2   Wread

Wread is a utility that uses the libweb100 library to periodically report TCP runtime variables maintained by web100 as requested by the user.

To list all current TCP connections in the local operating system: 

$ wread -l

To monitor a specific connection between the given source and destination IP addresses and ports and to report selected TCP runtime variables after each period specified in microseconds: 

wread <src ip> <src port> <dst ip> <dst port> \
     <time period> <var1> [<var2> ...]

With the above command you can make wread to wait for a future connection that satisfies specified parameters (such as a test connection started afterwards). This allows to capture the beginning of this connection, which is the most interesting part for performance debugging.

The output is then printed in the following format: 

Timestamp          <var1>   <var2>   ...  <varN>
<ts_value>         <value1> <value2> ...  <valueN>
   ...

For example, the following command can be used to wait for a connection from port 2000 on perfmon1.cesnet.cz to port 2001 on perfmon2.cesnet.cz and then sample PktsOut and PktsIn variables in 100 ms intervals: 

$ wread perfmon1.cesnet.cz 2000 perfmon2.cesnet.cz \
      2001 100000 PktsOut PktsIn

6.3   Tbwsyn

Tbwsyn utility extracts information from the output of different measurement tools according to instructions from the user and produces the data and command file for the gnuplot graphing utility.

The general syntax of using tbwsyn is the following: 

tbwsyn [ <general_parameters> ] [ -b<bulk_file> <bulk_parameters> ]
                                [ -w<wread_file> <wread_parameters> ]
                                [ -t<tcpdump_file> <tcpdump_parameters> ]

The most important general parameters that immediately follow the tbwsyn command are the following:

Time period specified by -c argument is used only for computed characteristics, such as average throughput computed from tcpdump tracefile. It does not affect the frequency of sampled characteristics, such as those obtained from web100 or bulk, which is determined at the time of measurement by arguments of wread and bulk utilities.

Sampling from web100 is specified exactly as a constant time period by an argument of the wread utility, whereas sampling from bulk is specified indirectly by the block size sent or received by one send() or recv() call and it can happen at varying frequency.

A time period specified by -p argument can be the same as the time period specified by -c argument or it can be different, particularly when we sampled web100 variables at one rate, computed tcpdump characteristics at a different rate and we want to plot all characteristics together at the finer of the two rates.

Parameters for bulk, wread and tcpdump are all specified in the same format as a comma-separated list of characteristics to report: 

-g<characteristics>[,<characteristics> ...]

The names of characteristics for bulk and wread are those accepted by these utilities, that is socket options and TCP runtime variables.

The names of characteristics for tcpdump, which are computed from the tracefile, can be the following:

The output of tbwsyn is in the format of a data file for gnuplot graphing utility: 

# bulk
# timestamp <colname1>   <colname2>       <colnameN>
<timestamp> <value1>     <value2>    ...  <valueN>
...

# wread
# timestamp <colname1>   <colname2>       <colnameN>
<timestamp> <value1>     <value2>    ...  <valueN>
...

# tcpdump (immediate characteristics)
# timestamp <colname1>   <colname2>       <colnameN>
<timestamp> <value1>     <value2>    ...  <valueN>
...

# tcpdump (computed characteristics)
# timestamp <colname1>   <colname2>       <colnameN>
<timestamp> <value1>     <value2>    ...  <valueN>
...

When -a argument is specified, timestamps are printed as absolute time in the format "HH:MM:SS uuuuuu", which includes hours, minutes, seconds and microseconds. Gnuplot can recognize time with granularity down to seconds (it cannot work with subsecond intervals). Therefore the absolute format is suitable only for long-term connections.

When -a argument is not specified, timestamps are printed relatively as the number of microseconds since the start of the connection. This is the format convenient to plot time evolution of short-term connections.

Tbwsyn is implemented in C++ with classes representing different protocols and monitored characteristics. Adding support for new characteristics is a relatively easy task.

6.4   Measure and gengraph scripts

The measure script provides an interface to all data-gathering tools. Particularly, it starts bulk, wread and tcpdump with appropriate command-line arguments to provide information requested by the user.

For example, assume that we have bulk running in the server mode on perfmon1.cesnet.cz. Then we can start a monitored test connection to this computer by the following command, which sets the block size to 16 kB and requests RTT monitoring from bulk and web100 and tcpdump monitoring: 

$ measure perfmon1.cesnet.cz -u 16384 -brtt -wSampleRTT -t

Note that we do not need to specify characteristics to be obtained from tcpdump, they can be selected later when using tbwsyn utility.

The gengraph script first calls tbwsyn utility to process and time-correlate outputs from the three data-gathering tools. Tbwsyn produces command and data files for standard graphing utility gnuplot. Gengraph then calls gnuplot with the generated command and data files.

For example, we can use the following command to request graphs showing RTT obtained from all three possible sources - bulk, web100 and tcpdump - and a graph showing average throughput computed from tcpdump, in the period between 2 and 3 seconds from the beginning of the connection with time granularity of 10 miliseconds: 

$ gengraph -p10000 -from 2000000 -to 3000000 \
   -brtt -wSampleRTT -trtt -tthroughput

The data flow between the tools started by the measure and gengraph scripts is illustrated in Figure.

[Figure]

Figure 4: Measure and Gengraph scripts

7   Implementation and operation notes

The tbwApplet is implemented in Java so that it can be used on workstations with various operating systems. The graphical user interface (GUI) is implemented using AWT (Abstract Windows Toolkit). We also needed to parse and construct XML messages for communication with tbwMP. One approach would be to use an existing SOAP implementation for Java, such as Apache Axis. However, this would add several megabytes to the applet downloaded by the web browser. Therefore, a simple parser of XML messages has been implemented in two threads - MeasureClient and GegraphClient, corresponding to the two phases of tbwApplet operation (see Section). The third thread called MeasureClient makes a test connection to the remote bulk tool.

The tbwMP measurement point is implemented in Python, as are several other MPs, which were developed for the perfSONAR system. We used ping MP as a basis to develop tbwMP. NMWGpy and SOAPpy packages are used to process NMWG-based XML messages. Remaining components were implemented in the C language.

The tbwMP measurement point must be started on a remote PC so that tbwApplet can connect to it. All other measurement tools and utilities are then started automatically by tbwMP. The following commands start tbwMP assuming that it is installed in the /usr/local/perfSONAR-MP-tbw directory and that the local IP address and the port number where tbwMP should listen for connections are 192.168.1.1 and 8090: 

$ cd /usr/local/perfSONAR-MP-tbw
$ ./service/bin/startup.sh -d -u 192.168.1.1 -p 8090 -l DEBUG

The control connection between tbwApplet and tbwMP and the test connection must pass through any firewalls that may be installed in a network path. You can choose a port number used by the control connection in the above command and you can choose a port number used by the test connection in the form presented by tbwApplet (this port number is passed over the control connection to the remote PC). If you start bulk manually in the server mode on some remote PC, you can also use a command-line argument for selecting the listening port number.

Default security policy for Java applets allows to use only port numbers starting from 1024 even for outgoing connections. If you want to use a lower port for the test connection, such as port 80, which is often permitted in firewalls, you need to change the Java security policy. In a Windows operating system, edit the following file (the exact Java version may differ): 

c:\Program Files\Java\jre1.5.0_06\lib\security\java.policy

and change the line that looks like this: 

permission java.net.SocketPermision "localhost:1024-", "listen";

to read: 

permission java.net.SocketPermision "localhost:1-", "listen";

TbwApplet offers the IP address of the local PC as the default destination IP address for the test connection from the remote PC. For security reasons, one of the operations that are prohibited for Java applets is finding out the IP addresses of local interfaces. To circumvent this restriction, the IP address of the user s workstation is obtained by tbwCGI script on the remote PC and sent back to tbwApplet.

We tested tbwtools with Debian Linux machines for tbwMP and the tools started from it and with Java JDK 1.4.2 for tbwApplet. More implementation details including the hierarchy of classes in the tbwsyn tool are described in [diplomka].

8   Example of use

A user starts TCP performance debugging by pointing a web browser to a web page where tbwApplet can be downloaded. An example installation of tbwtools is accessible for authorized users from http://perfmon.cesnet.cz.

The initial window presented by tbwApplet is shown in Figure. The user has two options.

The first option (1a) is to make a new test connection by specifying destination IP address, port number, direction, block size (to be sent in one send() call), socket buffer size and duration in seconds. The destination IP address field is pre-filled with the IP address of the user s PC. The user can make the test connection to the local PC or, by rewritting the IP address, to some remote PC running bulk in a server mode. The source IP address of the connection is that of the remote PC running tbwMP. In the current version, it is the PC from which tbwApplet was downloaded. A future version will allow to specify where tbwMP is running.

The second option (1b) is to enter data ID of an already performed test connection, whose measurement data is stored with tbwMP. This option can be used by an expert who is debugging performance problems on behalf of a user who makes a test connection and submit the data ID.

XML messages exchanged between tbwApplet and tbwMP can be observed in a frame below (2).

[Figure]

Figure 5: Doing test connection with tbwApplet

After the test connection finishes or the data ID is processed, the user is presented with a form that allows to select characteristics to be extracted from measurement data and presented, see Figure. The user can select any combination of variables obtained from tcpdump tracefile (3a), socket options from bulk monitoring (3b) or TCP runtime variables from web100 monitoring (3c). Finally, the user selects (4) the time period of the connection that should be extracted and the time intervals for computed variables and for plotting all characteristics (the latter also determines the interval for sampling variables from tcpdump tracefile).

[Figure]

Figure 6: Selecting characteristics with tbwApplet

An example of selected characteristics presented by tbwtools is shown in Figure. It shows, left to right and top to bottom:

In this particular connection, we can see that there were lots of packet losses, but only one retransmission timeout expiration, the rest of retransmissions were covered by the Fast Retransmit mechanism. Flat receiver window (rwin) shows that the receiver was not overloaded and steady availability of buffered not yet sent data shows that the sender was able to provide enough data to fill the network path. Overall it was a healthy connection, which was however competing for bandwidth all the time.

Tbwtools also presented the following summary about the connection: 

Transported 91226112 bytes in 10.796 s, connection speed 67.600 Mbps
There were 1146 packets retransmitted, 5097 duplicate ACKs received 
  and 0 SACK blocks received.
Segment size was 1448 B, average round-trip time 8.804 ms, 
  packet loss rate 0.090413%
The connection was receiver limited 1.68% of time
The connection was network limited 98.23% of time
Maximal window size of your machine was 384 kB which limits the throughput 
  to 357.899 Mbps
[Figure]

Figure 7: Example of characteristics plotted by tbwtools (large image)

9   Conclusion

We developed an integrated set of tools that provides a wide range of information for debugging TCP performance by combining TCP runtime variables and socket options with the connection tracefile.

Certainly, it is up to the user to evaluate what was happening during the test connection. In the future, a kind of expert system integrated with PERT knowledge base and case database [pert] can be developed to assist with this evaluation. What makes tbwtools particularly useful is the ability to get a wide range of time-correlated debugging information from different monitoring tools together in a common user interface. A user can easily change the set of produced characteristics and make more test connections with different parameters (such as socket buffer size or test duration) between different hosts. A user can also ask somebody else for the expert opinion by sharing the obtained data ID for an already made test connection and stored results.

We plan to integrate monitoring functionality of bulk into iperf, which would make it more accessible to people accustomed to iperf, a popular throughput measurement tool.

An online demo of tbwtools is accessible from https://perfmon.cesnet.cz. A public release of tbwtools is being prepared as part of JRA1 activity in the GN2 project [jra1].

10   Appendix. Messages for communication between tbwApplet and tbwMP

10.1   MetadataKeyRequest

An example of a message from tbwApplet to tbwMP in the first phase of performance debugging. Source and destination IP addresses of the test connection are specified, as well as the test duration, block size (to be sent in one send() call) and socket buffer size. 

<nmwg:message id="msg1"
      type="MetadataKeyRequest"
      xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/"
      xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"
      xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/"
      xmlns:ping="http://ggf.org/ns/nmwg/tools/tbw/2.0/">

  <nmwg:metadata id="meta1">

    <ping:subject id="sub1">
      <nmwgt:endPointPair>
        <nmwgt:src type="ipv4" value="10.43.70.6"/>
        <nmwgt:dst type="ipv4" value="10.43.70.5" port="2000"/>
      </nmwgt:endPointPair>
    </ping:subject>

    <nmwg:eventType>tbw</nmwg:eventType>

    <nmwg:parameters id="param1">
      <nmwg:parameter name="timeDuration">1</nmwg:parameter>
      <nmwg:parameter name="bufferLength">65536</nmwg:parameter>
      <nmwg:parameter name="windowSize">6000000</nmwg:parameter>
    </nmwg:parameters>

  </nmwg:metadata>

  <nmwg:data id="data1" metadataIdRef="meta1"/>

</nmwg:message>

10.2   MeasurementRequest

An example of a message from tbwApplet to tbwMP in the second phase of performance debugging. Source and destination IP addresses of an already done test connection are referenced. Then the lists of requested variables from bulk, web100 and tcpdump monitoring are specified. 

<nmwg:message id="msg1" type="MeasurementRequest"
      xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/"
      xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"
      xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/"
      xmlns:tbw="http://ggf.org/ns/nmwg/tools/tbw/2.0/">
       
  <nmwg:metadata id="meta1">

    <ping:subject id="sub1">
      <nmwgt:endPointPair>
        <nmwgt:src type="ipv4" value="10.43.70.6"/>
        <nmwgt:dst type="ipv4" value="10.43.70.5"/>
      </nmwgt:endPointPair>
    </ping:subject>

    <nmwg:eventType>tbw</nmwg:eventType>

    <nmwg:parameters id="param1">
      <nmwg:parameter name="interval">1</nmwg:parameter>
      <nmwg:parameter name="bulkVars">state,ca_state,retransmits
      </nmwg:parameter>
      <nmwg:parameter name="web100Vars">CurCwnd,CurSsthresh
      </nmwg:parameter>
      <nmwg:parameter name="tcpdumpVars">throughput</nmwg:parameter>
    </nmwg:parameters>

    <nmwg:key id="keymeta1">
     <nmwg:parameters>
      <nmwg:parameter name="datadir">fPcVH16825</nmwg:parameter>
     </nmwg:parameters>
    </nmwg:key>
   </nmwg:metadata>

   <nmwg:data id="data1" metadataIdRef="meta1"/>

</nmwg:message>

10.3   RELAX NG specification

A formal syntax of requests and responses exchanged between tbwApplet and tbwMP. 

namespace tbw = "http://ggf.org/ns/nmwg/tools/tbw/"
include "nmtopo.rnc"
include "nmbase.rnc" {
  Datum = TbwDatum
  Subject = TbwSubject
  Parameters = TbwParameters
}

## Tbw Metadata Section
TbwSubject = element tbw:subject {
  Identifier &
  EndpointPair
}

TbwParameters = element tbw:parameters {
  Identifier &
  (
    element tbw:timeDuration { xsd:int }? &
    element tbw:bufferLength { xsd:int }? &
    element tbw:windowSize { xsd:int }? &
    element tbw:direction { xsd:string }? & 
  ) |
  (
    element tbw:comPeriod { xsd:int }? &
    element tbw:sampleRate { xsd:int }? &
    element tbw:fromTime { xsd:int }? &
    element tbw:toTime { xsd:int }? &
    element tbw:bulkVars { xsd:string } &
    element tbw:web100Vars { xsd:string } &
    element tbw:tcpdumpVars { xsd:string }
  )
}

## Tbw Data Section
TbwDatum = element tbw:datum {
  attribute value { xsd:float } &
  attribute valueUnits { xsd:string }? &
  attribute numBytes { xsd:int }? &
  attribute numBytesUnits { xsd:string }? &
  attribute seqNum { xsd:int } &
}

References

[tcptrace] Tcptrace, http://jarok.cs.ohioe.edu/software/tcptrace.
[web100] Web100 Linux kernel instrumentation, http://www.web100.org.
[papi] PAPI - Performance Application Programming Interface, http://icl.cs.utk.edu/papi.
[perfsonar] perfSONAR monitoring framework, http://www.perfsonar.net.
[nmwg] NMWG - Network Measurement Working Group, http://nmwg.internet2.edu.
[bulk] Pavel Cimbál, Sven Ubik. Bulk utility for TCP performance debugging, CESNET Technical Report 29/2004.
[diplomka] Václav Řehák. Analysis of TCP Connections Behaviour with Regard to Communication Performance, Diploma Thesis, Czech Technical University, May 2006.
[pert] PERT - Performance Enhancement and Response Team, http://www.geant2.net/server/show/conWebDoc.999.
[jra1] perfSONAR infrastructure, http://www.perfsonar.net.
další weby:fond rozvojemetacentrumCzechLightpřenosyvideoservereduroameduID.cz