Not able to disable caching during fio I/O test
WillemDendauw opened this issue · comments
Description
I'm benchmarking gVisor and ran into confusing results when benchmarking its IO performance using fio. When running native or with runc, the latency is in the order of 6000ns, while gVisor achieves an average latency of around 100ns. The same goes for bandwidth, with a block size of 128Kb and an iodepth of 32, native and runc achieve around 360MiB/s while gVisor manages 2500MiB/s. Clearly this must be a caching issue. Luckily fio has an option to bypass caching '--direct=1', but apparently this doesn't work in the case of gVisor.
Now I have seen studies perform these tests and some do get the bandwidth in the right order of magnitude, but they claim that they can't fix the latency issue, and neither can I.
If someone could point me in the right direction that would we great.
Steps to reproduce
Run a pod on a Kubernetes cluster using runsc as the low-level container runtime and execute a fio benchmark from within that pod.
runsc version
runsc version release-20220425.0
spec: 1.0.2-devdocker version (if using docker)
using containerd: containerd containerd.io 1.5.11uname
5.4.0-67-generic #75-Ubuntu SMP Fri Feb 19 18:03:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
kubectl (if using Kubernetes)
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}repo state (if built from source)
No response
runsc debug logs (if available)
No response
A friendly reminder that this issue had no activity for 120 days.
What are the exact fio parameters you are using? I doubt this is a caching issue, since in runsc, regular files served by the gofer use a host file descriptor. We don't impose any page cache in the sentry.
A friendly reminder that this issue had no activity for 120 days.
This issue has been closed due to lack of activity.