device-virtual memory keeps increasing when device contains autoevents
cherrycl opened this issue · comments
🐞 Bug Report
Affected Services [REQUIRED]
The issue is located in: device-serviceIs this a regression?
No, Levski also has same issue.Description and Minimal Reproduction [REQUIRED]
- Deploy EdgeX by compose file
- Create a new device with us/ms interval autoevent, post device with request body as below.
[
{
"device":{
"name":"Test-Device",
"description":"Test-Device",
"adminState":"UNLOCKED",
"operatingState":"UP",
"protocols":{
"other":{
"Address":"simple01",
"Port":"300"
}
},
"labels":[
"device-sample"
],
"location":"{lat:45.45,long:47.80}",
"serviceName":"device-virtual",
"profileName":"Random-Integer-Device",
"autoEvents": [
{
"interval": "13us",
"onChange": false,
"sourceName": "Int8"
},
{
"interval": "10ms",
"onChange": false,
"sourceName": "Int16"
},
{
"interval": "1323us",
"onChange": false,
"sourceName": "Int32"
},
{
"interval": "331ms",
"onChange": false,
"sourceName": "Int64"
}
]
},
"apiVersion":"v3"
}
]
- Monitor container memory usage by
docker stats
🔥 Exception or Error
device-virtual memory usage is over than 100MB.
🌍 Your Environment
Deployment Environment: Ubuntu 2204 / x86
EdgeX Version [REQUIRED]: Main
Anything else relevant?
After running testing and analysis with the go pprof tool, the device service creates lots of goroutine to execute the auto event.
This issue can be resolved by:
- decreasing the auto event amount or interval to reduce the effort of device service
- increasing the Device.AsyncBufferSize to leverage the CPU capability to handle more auto event.
Device: AsyncBufferSize: 1
Another problem is the goroutine objects will not be removed from the system memory.
It's the Golang design, the Golang keeps the goroutine objects for reusing, so the memory will not be free.
related golang issue: golang/go#34457
Test 1 auto event with 100us and 1ms intervals. For 1ms, the both memory usages, device-virtual and app-rules-engine, still work fine. But for 100us, memory usage grows abnormal.
close this as a limitation