Openyurt using lot of bandwidth on edge nodes while communicating to cloud node
nics90 opened this issue · comments
What happened:
We created 5 node cluster with 3 masters nodes running in cloud and 2 worker nodes running on Edge devices. Our Edge Devices uses 4G network to connect to internet, hence edge to cloud communication occur on that network. When we monitored the data usage on the 4G router we found that yurthub is consuming lot of data (100 MB per min), which means in a days it consuming approx 100 GB of data in 24 hours.
What you expected to happen:
There is something wrong here since it should not consume that much of data. Is there any configuration to reduce yurthub data consumption.
How to reproduce it (as minimally and precisely as possible):
Install OpenYurt Version 1.4.0 and Raven 0.4.0 and monitor the traffic from edge node to cloud node, it gets replicated.
Anything else we need to know?:
Environment:
- OpenYurt version: 1.4.0
- Kubernetes version (use
kubectl version): 1.24 - OS (e.g:
cat /etc/os-release): Ubuntu 22.04.3 LTS - Kernel (e.g.
uname -a): Linux jfl-edgebox-01 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux - Install tools:
- Others:
others
/kind question
@nics90 Thanks for raising the issue. Would you like to upload the metrics of yurthub component? and you can execute curl http://127.0.0.1:10267/metrics command on the edge node to get metrics of yurthub component.
The response traffic data is recorded in the metrics as request level.
@rambohe-ch : Please find the attached files containing the metrics captured from two worker nodes.
worker1.txt
worker2.txt
@nics90 As the above metrics, there are too many requests that are sent by clients(such as longhorn-manager, go-http-client, calico-node, etc.)on worker nodes. and some requests are rejected by yurthub because of threshold is over.
@nics90 unfortunately, the traffic of these clients is not recorded in the metrics, you need to open cache for these agents in Yurthub.
@rambohe-ch : Could you please help here and let me know how to open cache for these clients ? Thanks a lot!
@rambohe-ch : Could you please help here and let me know how to open cache for these clients ? Thanks a lot!
@nics90 you can configure cache_agents field as * in yurt-hub-cfg configmap in order to enable yurthub cache response for all agents.
after you modify the above configuration, you need recreate all the pods on the node in order to construct cache from the beginning.
@rambohe-ch : I tried the above config and it started caching all the clients, but unfortunately instead of decreasing the data usage it increased the usage. Could you please help here on priority basis.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.