Device Communication Aborted for devices having address in format "<RemoteStation 51:17>"
Zedstron opened this issue · comments
Describe the bug
Well, I am not sure if it's a bug or not but after scanning the devices using bacnet_scan.py, there are two types of devices are scanned one category has the following template.
Device Address = <Address 192.168.1.26>
Device Id = 66617
maxAPDULengthAccepted = 1476
segmentationSupported = segmentedBoth
vendorID = 3
While the other category falls under the following template
Device Address = <RemoteStation 51:17>
Device Id = 171700
maxAPDULengthAccepted = 286
segmentationSupported = noSegmentation
vendorID = 11
The problem is I can get data e.g., read_properties work fine for the devices following first template but not for the RemoteStation thingy. The error is following.
2023-07-10 12:15:12,001 (platform_driveragent-4.0 1255) platform_driver.driver DEBUG: my_campus/my_building/floor1 next scrape scheduled: 2023-07-10 07:20:12+00:00
2023-07-10 12:15:12,002 (platform_driveragent-4.0 1255) platform_driver.driver DEBUG: scraping device: my_campus/my_building/floor1
2023-07-10 12:15:12,013 (bacnet_proxyagent-0.5 1221) __main__ DEBUG: Reading 11 points on 50:14, max per scrape: 10
2023-07-10 12:15:12,015 (bacnet_proxyagent-0.5 1221) __main__ DEBUG: Requesting 10 properties from 50:14
2023-07-10 12:15:19,238 (listeneragent-3.3 504) __main__ INFO: Peer: pubsub, Sender: config.store:, Bus: , Topic: heartbeat/config.store, Headers: {'TimeStamp': '2023-07-10T07:15:19.233048+00:00', 'min_compatible_version': '3.0', 'max_compatible_version': ''}, Message:
'GOOD'
2023-07-10 12:15:21,111 (bacnet_proxyagent-0.5 1221) volttron.platform.vip.agent.subsystems.rpc ERROR: unhandled exception in JSON-RPC method 'read_properties':
Traceback (most recent call last):
File "/code/volttron/volttron/platform/vip/agent/subsystems/rpc.py", line 158, in method
return method(*args, **kwargs)
File "/home/volttron/.volttron/agents/c30ba184-74bd-4671-9d07-4f0a83966dae/bacnet_proxyagent-0.5/bacnet_proxy/agent.py", line 770, in read_properties
bacnet_results = iocb.ioResult.get(10)
File "src/gevent/event.py", line 329, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 359, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 347, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 327, in gevent._gevent_cevent.AsyncResult._raise_exception
File "/home/volttron/.local/lib/python3.7/site-packages/gevent/_compat.py", line 67, in reraise
raise value
RuntimeError: Device communication aborted: noResponse
2023-07-10 12:15:21,120 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: ERROR:volttron.platform.vip.agent.subsystems.rpc:unhandled exception in JSON-RPC method 'read_properties':
2023-07-10 12:15:21,121 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: Traceback (most recent call last):
2023-07-10 12:15:21,122 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: File "/code/volttron/volttron/platform/vip/agent/subsystems/rpc.py", line 158, in method
2023-07-10 12:15:21,122 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: return method(*args, **kwargs)
2023-07-10 12:15:21,124 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: File "/home/volttron/.volttron/agents/c30ba184-74bd-4671-9d07-4f0a83966dae/bacnet_proxyagent-0.5/bacnet_proxy/agent.py", line 770, in read_properties
2023-07-10 12:15:21,125 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: bacnet_results = iocb.ioResult.get(10)
2023-07-10 12:15:21,125 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: File "src/gevent/event.py", line 329, in gevent._gevent_cevent.AsyncResult.get
2023-07-10 12:15:21,126 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: File "src/gevent/event.py", line 359, in gevent._gevent_cevent.AsyncResult.get
2023-07-10 12:15:21,126 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: File "src/gevent/event.py", line 347, in gevent._gevent_cevent.AsyncResult.get
2023-07-10 12:15:21,127 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: File "src/gevent/event.py", line 327, in gevent._gevent_cevent.AsyncResult._raise_exception
2023-07-10 12:15:21,127 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: File "/home/volttron/.local/lib/python3.7/site-packages/gevent/_compat.py", line 67, in reraise
2023-07-10 12:15:21,128 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: raise value
2023-07-10 12:15:21,129 (bacnet_proxyagent-0.5 1221) <stderr> ERROR: RuntimeError: Device communication aborted: noResponse
2023-07-10 12:15:21,129 (bacnet_proxyagent-0.5 1221) <stderr> ERROR:
2023-07-10 12:15:21,120 (platform_driveragent-4.0 1255) platform_driver.driver ERROR: Failed to scrape my_campus/my_building/floor1:
Traceback (most recent call last):
File "/home/volttron/.volttron/agents/74cbfe1f-21d9-485f-9741-9be3e86a6298/platform_driveragent-4.0/platform_driver/driver.py", line 244, in periodic_read
results = self.interface.scrape_all()
File "/home/volttron/.volttron/agents/74cbfe1f-21d9-485f-9741-9be3e86a6298/platform_driveragent-4.0/platform_driver/interfaces/bacnet.py", line 164, in scrape_all
self.max_per_request, self.use_read_multiple).get(timeout=self.timeout)
File "src/gevent/event.py", line 329, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 359, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 329, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 329, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 347, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 327, in gevent._gevent_cevent.AsyncResult._raise_exception
File "/home/volttron/.local/lib/python3.7/site-packages/gevent/_compat.py", line 67, in reraise
raise value
volttron.platform.jsonrpc.RemoteError: builtins.RuntimeError('Device communication aborted: noResponse')
To Reproduce
Use any device to read data which has no IP address but represented as a RemoteStation, simply scan the devices which has atleast one RemoteStation involved, Import the scanned configurations and check logs
Expected behavior
The device should be scraped successfully.
Operating System
- OS: Ubuntu 18.04
- Volttron Version: volttron/volttron:develop,
Additional context
The devices are accessible in building BMS and showing values, so its hardly possible that they are not responding, Tried in two different buildings, In one site it does not read at all as shown above , in one building it reads the data for about 2 to 3 hours and then starts throwing same exact error unless the BACnet proxy agent is restarted, I am not able to conclude the root cause, as it's a strange behavior.
@rlutes @davidraker @acedrew any thoughts on this?
This looks like you might be trying to readPropertyMultiple on a device that doesn't support it, have you tried using read_property_multiple: false in the device config?
I don't have access to the site remotely, today I have a site visit I will try this and let you know here
I noticed volttron works fine over BACnet/IP but for BACnet/MSTP it throws errors and the communication is not reliable
Finally i kept the sugested property with false value in the config, now my config looks like following
{
"driver_config": {
"device_address": "50:7",
"device_id":70700,
"read_property_multiple": false
},
"driver_type": "bacnet",
"registry_config": "config://registry_configs/50:7_70700.registry",
"interval":300
}
And the error is now changed 30 seconds Timeout, note that there are almost 20+ properties to read in the device
I just discovered the root cause, its the bacpypes version which is too old in volttron container, following code confirms it
# Code which works using bacpypes==0.18.1 and not with 0.16.7
import BAC0
bacnet = BAC0.lite(ip="192.168.1.20/24", port=47808)
def main():
payload = "50:7 analogOutput 1 presentValue"
print("Requesting {}".format(payload))
print("The value is ", bacnet.read(payload))
if __name__ == '__main__':
main()
The same device is returning values using this code but Communication aborted in volttron container
That's not a root cause. There's no confirmation in your example. You're comparing reading one point, once with no timing information, with reading 20 points in 30 seconds.
It sounds like you're dealing with an especially old and slow device. Try reducing the number of points, or increasing the timeout. You may also need to increase the request interval to give the device time to respond.
You state that you have communication aborted after confirming you have timeout.
The sample code you provide is using readProperty, not readPropertyMultiple, under the hood.
So, after a lot of testing and efforts, you are right the previous code was using single property read, now I changed the script to read all objects with 0.5 seconds of delay between each read and called readMultiple for each object to read all properties at once, Now following are my test results.
- Volttron showed same error for single property read.
- Increasing the timeout to 100 (as you said it's a slow device) didn't worked either.
Regarding the code to test whether the device really doesn't support readMultiple and is really that much slow to require more then Volttron''s default 30s timeout, I used following version.
import BAC0
import json
from time import time, sleep
from tqdm import tqdm
bacnet = BAC0.lite(ip="192.168.1.20/24", port=47808)
attributes = json.load(open('devices/deviceB.json', 'r'))
print("Total Objects Found: {}".format(len(attributes)))
def main():
total_objects = len(attributes)
pbar = tqdm(total=total_objects, unit='object(s)')
start_time = time()
for attribute in tqdm(attributes, desc="Requesting Objects from RemoteStation"):
payload = "{} {} {}".format(attribute['BACnet Object Type'], attribute['Index'], 'all')
payload = "50:7 " + payload
value = bacnet.readMultiple(payload)
attribute["Value"] = value
sleep(0.5)
elapsed_time = time() - start_time
print("Total time took: {:02d}:{:02d}".format(int(elapsed_time // 60), int(elapsed_time % 60)))
if __name__ == '__main__':
main()
There is total 41 actual objects in the Json file I was loading, so baseline delay is 41 x 0.5 = 20.5s and the total script execution time after including actual reading delay was 28 seconds which is still less then default Voltron timeout value 30.
I request you to provide me a test case compatible device config which you think might work on slow device maybe I am missing some configurations.
The screenshots of the logs are following
and
I agree root cause is not related to bacpypes version, but there is something else which need attention regarding slow devices more specifically regarding MS/TP devices.
I will be very thankful to you @acedrew for assisting me to solve this issue, as by solving this problem you will solve a very big problem of my life 😟
Closing this issue after a complete diagnosis, extensive debugging and analysis of the operating system itself, I am closing this issue as it's NOT THE PROBLEM WITH VOLTTRON itself but it's a strange behavior of Ubuntu 20.04 which when goes idle after some time, changes the routing table and most of the network devices are not accessible anymore unless we login back to device.