Apache Benchmark Performace is slow
agdomacena2 opened this issue · comments
Hi, I was comparing the performance of the apache_benchmark (ab) that came with mTCP with the current ab that is available from Apache. I noticed that the mTCP ab is slower and does less requests per second on average (12 seconds on 100 requests and 100 concurrency with 7 requests per second mean compared to 0.209 seconds on 100 requests and 100 concurrency with 470 requests per second mean) I already tried leaving only 1 core online during tests and the results are the same.
The server I used is the the python3 http.server. The mTCP machine and the server are connected directly instead of using a switch.
For your paper, did you use your own version of ab configured without mtcp and dpdk and compared them to one that is configured with mtcp and dpdk?
could you share your ab running example test output? the mTCP ab has multithread support, did you run with mutiple core?
We did not test ab-mtcp's performance with as little as 100 concurrent connections. For lower concurrency values, the batching effect may have a detrimental effect on latency numbers. Having said that, I still think that the flow completion times are really low for mTCP's. Can you try running the same experiment with at least 1000 concurrent flows? If you still get poor results, then I would know that there must be something wrong with your settings. In the mean time, I will try to find some time and conduct similar experiments on my side as well.
Hi, I tried the experiment again with 10,000 requests and 1,000 concurrent connections for both ab and ab-mtcp when both are running on only 1 core (only 1 core is online) and my server is now a simple lighttpd server instead of the python server. I used the following commands for ab and ab-mtcp.
ab: ab -n 10000 -c 1000 10.0.0.2:8889/rawr.txt
ab-mtcp: sudo ./ab -N 1 -n 10000 -c 1000 10.0.0.2:8889/rawr.txt
ab has an average of 5.3518 after it experienced a timeout (recorded at 33 seconds) with an average of 4000 requests per second and an average transfer rate of 1000 kBytes/sec
ab-mtcp has an average of 5.8213 with an average of 1,800 requests per second and an average transfer rate of around 500 kBytes/sec
It seems that the transfer rate of ab-mtcp is the biggest difference and that the ab-mtcp program continues even after the 10,000 requests are completed forcing me to stop it using ctrl+c whenever I see the Finished 10000 requests in the command terminal.
Below are the ab running example test output
AB
AB-mTCP
I am not sure if the AB-mTCP statistics output are accurate, is there any other way you can measure the throughput?
Hi, after checking with wireshark (I captured at the server side) the reported time taken of ab-mtcp does not coincide with the difference between the first packet for the connection and the last packet in wireshark. The time it takes for 1,000 concurrent connections and 10,000 requests as reported by ab-mtcp is 6.53 seconds (this is a new run) and the time from first packet for the connection and last packet reported by wireshark is 3.515 seconds.
Hi, @ajamshed
I found the reason why ab-mtcp is reporting slower numbers. It adds the initialization time of the mtcp thread context and the initialization is the reason why ab-mtcp is slower for lower than 100,000 requests. If you do not add the initialization time, ab-mtcp is faster and if you use more than 100,000 requests then the time taken by the initialization of the thread is buried by the time taken for the test itself therby reporting more accurate numbers.