Tweaked iperf3 to use MPTCP socket API

This branch contains patches for iperf3 to smartly control MPTCP tests with socket API.

Below commands require to be run on an MPTCP-enabled Linux and with the patches for enhanced MPTCP socket API by Benjamin Hesmans.

Server side

On server, just create iperf server as usual:

$ src/iperf3 -s

We can also specify mptcp scheduler on server:

$ sudo src/iperf3 -s --scheduler roundrobin

By definition, it will apply on traffic from server to client only. Notice that the redundant scheduler is currently unstable and can cause system crash.

Client side

On client, we can specify a list of subflows with option -m or --subflows

by interface names:

$ src/iperf3 -c 37.187.114.89  -m eth0,eth1

or by client's IP addresses:

$ src/iperf3 -c 37.187.114.89  -m 192.168.56.101,192.168.56.101

or a combination of interface and IP address:

$ src/iperf3 -c 37.187.114.89  -m eth1,192.168.56.101

or a combination of IPv4 and IPv6:

$ src/iperf3 -c 2001:41d0:a:6759::1  -m  2001:6a8:3081:4f00:a00:27ff:fec4:285c,192.168.56.101

We can also specify mptcp scheduler on client:

$ sudo src/iperf3 -c 37.187.114.89  -m eth0,eth1  --scheduler roundrobin

Output

On client side, the statistics of all subflows are printed each interval. If json (-J) is specified, the program also writes output to files in results/ folder, in parallel to user-friendly output on stdout. This is different from original iperf3, which only outputs either user-friendly or json output.

On server side, after each test finished, the test results are collected from client and are stored in a json file in current directory. Its name format is:

./iperf_test_server_isReceiver_1479420880.json

which contains results of each subflow for each interval, and metadata of the test.

Backward compatible

There is a parameters exchange phase between client and server before actual test, so an enhanced iperf instance can know whether its peer is a modified or original one. If it knows that the peer is original one, it will fallback to regular iperf behavior.

During this phase, they also exchange all their local IP addresses, so that they can create subflows on multiple IP addresses, even when only a single remote server address is specified.

It is noticed that this subflow argument does not co-exist with parallel TCP streams argument (-P). You can only use either (-m) or (-P) for a test.

Below is the original iperf3 documentation.

iperf3: A TCP, UDP, and SCTP network bandwidth measurement tool

Summary

iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks. It supports tuning of various parameters related to timing, protocols, and buffers. For each test it reports the bandwidth, loss, and other parameters.

This version, sometimes referred to as iperf3, is a redesign of an original version developed at NLANR/DAST. iperf3 is a new implementation from scratch, with the goal of a smaller, simpler code base, and a library version of the functionality that can be used in other programs. iperf3 also a number of features found in other tools such as nuttcp and netperf, but were missing from the original iperf. These include, for example, a zero-copy mode and optional JSON output. Note that iperf3 is not backwards compatible with the original iperf.

Primary development for iperf3 takes place on CentOS Linux, FreeBSD, and MacOS X. At this time, these are the only officially supported platforms, however there have been some reports of success with OpenBSD, NetBSD, Android, Solaris, and other Linux distributions.

iperf3 is principally developed by ESnet / Lawrence Berkeley National Laboratory. It is released under a three-clause BSD license.

Note that at this point, ESnet plans to support iperf3 in "maintenance mode". At this point, there are no definite plans for further iperf3 releases, and ESnet will be providing a very limited amount of resources for support and development, going forward. However, ESnet could issue new iperf3 releases to deal with security issues or high-impact bug fixes.

For more information see: http://software.es.net/iperf

Source code and issue tracker: https://github.com/esnet/iperf

Obtaining iperf3

Downloads of iperf3 are available at:

http://downloads.es.net/pub/iperf/

To check out the most recent code, clone the git repository at:

https://github.com/esnet/iperf.git

Building iperf3

Prerequisites:

None.

Building

./configure; make; make install

(Note: If configure fails, try running ./bootstrap.sh first) If the error "libiperf.so not found" happens when run ./iperf3, update LD_LIBRARY_PATH with:

sudo ldconfig /usr/local/lib

Invoking iperf3

iperf3 includes a manual page listing all of the command-line options. The manual page is the most up-to-date reference to the various flags and parameters.

For sample command line usage, see:

http://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf-and-iperf3/

Using the default options, iperf is meant to show typical well designed application performance. 'Typical well designed application' means avoiding artificial enhancements that work only for testing (such as splice()'ing the data to /dev/null). iperf does also have flags for 'extreme best case' optimizations but they must be explicitly activated.

These flags include:

-Z, --zerocopy            use a 'zero copy' sendfile() method of sending data
-A, --affinity n/n,m      set CPU affinity

Bug Reports

Before submitting a bug report, please make sure you're running the latest version of the code, and confirm that your issue has not already been fixed. Then submit to the iperf3 issue tracker on GitHub:

https://github.com/esnet/iperf/issues

In your issue submission, please indicate the version of iperf3 and what platform you're trying to run on (provide the platform information even if you're not using a supported platform, we might be able to help anyway). Exact command-line arguments will help us recreate your problem. If you're getting error messages, please include them verbatim if possible, but remember to sanitize any sensitive information.

If you have a question about usage or about the code, please do not submit an issue. Please use one of the mailing lists for that.

Changes from iperf 2.x

New options:

-V, --verbose             more detailed output than before
-J, --json                output in JSON format
-Z, --zerocopy            use a 'zero copy' sendfile() method of sending data
-O, --omit N              omit the first n seconds (to ignore slowstart)
-T, --title str           prefix every output line with this string
-F, --file name           xmit/recv the specified file
-A, --affinity n/n,m      set CPU affinity (Linux and FreeBSD only)
-k, --blockcount #[KMG]   number of blocks (packets) to transmit (instead 
                          of -t or -n)
-L, --flowlabel           set IPv6 flow label (Linux only)

Changed flags:

-C, --linux-congestion    set congestion control algorithm (Linux only)
                          (-Z in iperf2)

Deprecated options:

Not planning to support these iperf2 flags. If you really miss these options, please submit a request in the issue tracker:

-d, --dualtest           Do a bidirectional test simultaneously
-r, --tradeoff           Do a bidirectional test individually
-T, --ttl                time-to-live, for multicast (default 1)
-x, --reportexclude [CDMSV]   exclude C(connection) D(data) M(multicast) 
                              S(settings) V(server) reports
-y, --reportstyle C      report as a Comma-Separated Values

Also deprecated is the ability to set the options via environment variables.

Known Issues

UDP performance: Some problems have been noticed with iperf3 on the ESnet 100G testbed at high UDP rates (above 10Gbps). The symptom is that on any particular run of iperf3 the receiver reports a loss rate of about 20%, regardless of the -b option used on the client side. This problem appears not to be iperf3-specific, and may be due to the placement of the iperf3 process on a CPU and its relation to the inbound NIC. In some cases this problem can be mitigated by an appropriate use of the CPU affinity (-A) option. (Issue #55)
The -Z flag sometimes causes the iperf3 client to hang on OSX. (Issue #129)
When specifying the socket buffer size using the "-w" flag on Linux, Linux doubles the value you pass in. (You can see this using iperf3's debug flag). But then the CWND does not actually ramp up to the doubled value, but only to about 75% of the doubled value. This appears to be by design.
Although the "-w" flag is documented as setting the (TCP) window size, it is also used to set the socket buffer size. This has been shown to be helpful with high-bitrate UDP tests.
On some platforms, it might be necessary to invoke "ldconfig" manually after doing a "make install" before the iperf3 executable can find its shared library. (Issue #153)
The results printed on the server side at the end of a test do not correctly reflect the client-side measurements. This is due to the ordering of computing and transferring results between the client and server. (Issue #293)
The server could have a very short measurement reporting interval at the end of a test (particularly a UDP test), containing few or no packets. This issue is due to an artifact of timing between the client and server. (Issue #278)

Copyright

iperf, Copyright (c) 2014-2016, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Technology Transfer Department at TTD@lbl.gov.

NOTICE. This software is owned by the U.S. Department of Energy. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, prepare derivative works, and perform publicly and display publicly. Beginning five (5) years after the date permission to assert copyright is obtained from the U.S. Department of Energy, and subject to any subsequent five (5) year renewals, the U.S. Government is granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, prepare derivative works, distribute copies to the public, perform publicly and display publicly, and to permit others to do so.

This code is distributed under a BSD style license, see the LICENSE file for complete information.

hoang-tranviet / iperf-siri