Giters

nathanmarlor / foxess_modbus

FoxESS inverter integration. Connect directly to your FoxESS inverter (no cloud!) for real-time status and control.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Serial connections fail when there are many open fds

djdarlo opened this issue · comments

commented

Version of the custom_component

Configuration

Add your logs here.

Describe the bug

A clear and concise description of what the bug is.

I am using the modbus serial integration. Every so often the sensors stop reporting. A simple restart of HA sorts it but I have noticed these errors in the log.

Debug log


Add your logs here.

This error originated from a custom integration.

Logger: custom_components.foxess_modbus.modbus_controller
Source: custom_components/foxess_modbus/modbus_controller.py:220
Integration: FoxESS - Modbus (documentation, issues)
First occurred: 06:35:45 (1 occurrences)
Last logged: 06:35:45

/dev/ttyUSB0 247 - 5 failed poll attempts: now not connected. Last error: filedescriptor out of range in select()


This error originated from a custom integration.

Logger: custom_components.foxess_modbus.modbus_controller
Source: custom_components/foxess_modbus/modbus_controller.py:152
Integration: FoxESS - Modbus (documentation, issues)
First occurred: 06:34:57 (800 occurrences)
Last logged: 09:54:46

General exception when polling /dev/ttyUSB0 247: ValueError('filedescriptor out of range in select()')
Traceback (most recent call last):
  File "/config/custom_components/foxess_modbus/modbus_controller.py", line 152, in _refresh
    reads = await self._client.read_registers(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/foxess_modbus/modbus_client.py", line 210, in read_registers
    response = await self._async_pymodbus_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/foxess_modbus/modbus_client.py", line 298, in _async_pymodbus_call
    result = await self._hass.async_add_executor_job(call, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/mixin.py", line 116, in read_input_registers
    return self.execute(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/base.py", line 192, in execute
    return self.transaction.execute(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 159, in execute
    response, last_exception = self._transact(
                               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 275, in _transact
    size = self._send(packet)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 323, in _send
    return self.client.framer.sendPacket(packet)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/framer/rtu_framer.py", line 296, in sendPacket
    size = self.client.send(message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/serial.py", line 278, in send
    size = self.socket.write(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/serial/serialposix.py", line 640, in write
    abort, ready, _ = select.select([self.pipe_abort_write_r], [self.fd], [], None)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: filedescriptor out of range in select()
commented

Looks the same as #273, but for USB devices rather than network devices.

The problem is that you have too many file descriptors open (more than 1024), and Python's serial library can't cope.

I'm not honestly sure what we're going to be able to do about it, as the problematic code is two layers of abstraction below us.

commented

Hmm, looks like pyserial has a PosixPollSerial class, which will do what we want. Trying to work out how to get pymodbus to use it...

commented

https://pyserial.readthedocs.io/en/latest/url_handlers.html#alt. I'll try and give that a try over the weekend, although I'll need some help testing as I don't have a serial USB device to test with.

commented
commented

Reopening until we've confirmed this works

commented

@djdarlo Would you mind trying out the pre-release v1.8.0b3, see if that fixes it for you? Thanks!

commented
commented
commented

Hacs - Integrations - FoxESS Modbus - Hamburger menu in the top right - Redownload, then select the "Show beta versions" toggle

commented
commented

Thanks!

commented

The other one errored yesterday so I've updated that one too. Both going fine so far. I'll keep you updated. Thank you.

commented

I spoke too soon. One of my HA installs went pop over night :( Other one is still going fine. Logs below...

This error originated from a custom integration.

Logger: custom_components.foxess_modbus.modbus_controller
Source: custom_components/foxess_modbus/modbus_controller.py:129
Integration: FoxESS - Modbus (documentation, issues)
First occurred: November 5, 2023 at 8:30:16 PM (1 occurrences)
Last logged: November 5, 2023 at 8:30:16 PM

Aborting refresh of /dev/ttyUSB0 247 as a previous refresh is still in progress. Is your poll rate '15' too high?

This error originated from a custom integration.

Logger: custom_components.foxess_modbus.modbus_controller
Source: custom_components/foxess_modbus/modbus_controller.py:153
Integration: FoxESS - Modbus (documentation, issues)
First occurred: 1:45:24 AM (3 occurrences)
Last logged: 1:47:24 AM

General exception when polling /dev/ttyUSB0 247: UnboundLocalError("cannot access local variable 'fd' where it is not associated with a value")
Traceback (most recent call last):
  File "/config/custom_components/foxess_modbus/modbus_controller.py", line 153, in _refresh
    reads = await self._client.read_registers(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/foxess_modbus/modbus_client.py", line 218, in read_registers
    response = await self._async_pymodbus_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/foxess_modbus/modbus_client.py", line 306, in _async_pymodbus_call
    result = await self._hass.async_add_executor_job(call, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/mixin.py", line 117, in read_input_registers
    pdu_reg_read.ReadInputRegistersRequest(address, count, slave, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/base.py", line 177, in execute
    self.delay_ms = self.params.reconnect_delay
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 168, in execute
    )
      
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 313, in _transact
    ) as msg:
              
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 392, in _recv
    result = read_min + result
             ~~~~~~~~~^~~~~~~~~
  File "/usr/local/lib/python3.11/site-packages/pymodbus/framer/rtu_framer.py", line 296, in recvPacket
    size = self.client.send(message)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/serial.py", line 271, in recv
    return result
  File "/usr/local/lib/python3.11/site-packages/serial/serialposix.py", line 830, in read
    if fd == self.pipe_abort_read_r:
       ^^
UnboundLocalError: cannot access local variable 'fd' where it is not associated with a value

This error originated from a custom integration.

Logger: custom_components.foxess_modbus.modbus_controller
Source: custom_components/foxess_modbus/modbus_controller.py:221
Integration: FoxESS - Modbus (documentation, issues)
First occurred: 1:50:23 AM (1 occurrences)
Last logged: 1:50:23 AM

/dev/ttyUSB0 247 - 5 failed poll attempts: now not connected. Last error: filedescriptor out of range in select()

Logger: pymodbus.logging
Source: /usr/local/lib/python3.11/site-packages/pymodbus/logging.py:108
First occurred: 1:45:21 AM (1181 occurrences)
Last logged: 6:44:01 AM

Cleanup recv buffer before send: 0xf7 0x4 0xba 0x5 0x88 0x0 0x0 0x0 0x4 0x3 0xb9 0x0 0x0 0xff 0xfe 0x9 0x38 0x0 0x17 0x2 0x2a 0x9 0x72 0x0 0x17 0x2 0x14 0xff 0xd7 0x2 0x34 0x13 0x92 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xff 0xf7 0x0 0x0 0x2 0x21 0x1 0x1a 0x1 0x78 0x0 0x0 0x0 0x0 0x17 0x70 0xe8 0x90 0x0 0x0 0x0 0x0 0x2 0x2 0x5 0x5 0x9 0x35 0xff 0xe8 0x0 0x2f 0x2 0x28 0x0 0xe5 0xa 0x44 0x7 0x8 0x1 0xf4 0x1 0xf4 0x0 0xf1 0x0 0xb9 0xc 0xc7 0xc 0xb9 0x4 0xf 0x0 0xb2 0xe0 0xb 0x0 0xa 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x53 0x0 0x0 0x0 0x2 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xcc 0x58 0x0 0x0 0x0 0x0 0x4d 0xad 0x0 0x1c 0x0 0x0 0x48 0x6a 0x0 0x10 0x0 0x0 0x4b 0xd6 0x0 0x0 0x0 0x0 0x31 0xde 0x0 0x1f 0x0 0x0 0xcd 0x53 0x0 0xe 0x0 0x0 0x19 0xb1 0x0 0x1e 0x0 0x0 0x97 0x82 0x0 0xf 0x95 0x95
Cleanup recv buffer before send: 0xf7 0x4 0xba 0x5 0x79 0x0 0x0 0x0 0x5 0x3 0xc3 0x0 0x0 0xff 0xff 0x9 0x38 0x0 0x17 0x2 0x2c 0x9 0x73 0x0 0x17 0x2 0x11 0xff 0xd9 0x2 0x32 0x13 0x90 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xff 0xee 0x0 0x0 0x2 0x23 0x1 0x1a 0x1 0x78 0x0 0x0 0x0 0x0 0x17 0x70 0xe8 0x90 0x0 0x0 0x0 0x0 0x2 0x2 0x5 0x5 0x9 0x35 0xff 0xe7 0x0 0x2f 0x2 0x27 0x0 0xe5 0xa 0x44 0x7 0x8 0x1 0xf4 0x1 0xf4 0x0 0xf1 0x0 0xb9 0xc 0xc6 0xc 0xb9 0x4 0xf 0x0 0xb2 0xe0 0xd 0x0 0xa 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x53 0x0 0x0 0x0 0x2 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xcc 0x58 0x0 0x0 0x0 0x0 0x4d 0xad 0x0 0x1c 0x0 0x0 0x48 0x6a 0x0 0x10 0x0 0x0 0x4b 0xd6 0x0 0x0 0x0 0x0 0x31 0xde 0x0 0x1f 0x0 0x0 0xcd 0x53 0x0 0xe 0x0 0x0 0x19 0xb1 0x0 0x1e 0x0 0x0 0x97 0x82 0x0 0xf 0x8e 0x9a
Cleanup recv buffer before send: 0xf7 0x4 0xba 0x5 0x9c 0x0 0x0 0x0 0x4 0x3 0xcc 0x0 0x0 0xff 0xfd 0x9 0x38 0x0 0x17 0x2 0x27 0x9 0x73 0x0 0x17 0x2 0x10 0xff 0xd7 0x2 0x31 0x13 0x8d 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xff 0xfe 0x0 0x0 0x2 0x12 0x1 0x1a 0x1 0x78 0x0 0x0 0x0 0x0 0x17 0x70 0xe8 0x90 0x0 0x0 0x0 0x0 0x2 0x2 0x5 0x5 0x9 0x35 0xff 0xe8 0x0 0x2f 0x2 0x27 0x0 0xe5 0xa 0x44 0x7 0x8 0x1 0xf4 0x1 0xf4 0x0 0xf1 0x0 0xb8 0xc 0xc6 0xc 0xba 0x4 0xf 0x0 0xb2 0xe0 0x10 0x0 0xa 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x53 0x0 0x0 0x0 0x2 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xcc 0x58 0x0 0x0 0x0 0x0 0x4d 0xad 0x0 0x1c 0x0 0x0 0x48 0x6a 0x0 0x10 0x0 0x0 0x4b 0xd6 0x0 0x0 0x0 0x0 0x31 0xde 0x0 0x1f 0x0 0x0 0xcd 0x53 0x0 0xe 0x0 0x0 0x19 0xb1 0x0 0x1e 0x0 0x0 0x97 0x82 0x0 0xf 0xc6 0x9b
Cleanup recv buffer before send: 0xf7 0x4 0xba 0x5 0x9f 0x0 0x0 0x0 0x4 0x3 0xd4 0x0 0x0 0x0 0x0 0x9 0x38 0x0 0x17 0x2 0x27 0x9 0x6c 0x0 0x17 0x2 0xc 0xff 0xdd 0x2 0x2c 0x13 0x8e 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xff 0xf1 0x0 0x0 0x2 0x1c 0x1 0x1a 0x1 0x78 0x0 0x0 0x0 0x0 0x17 0x70 0xe8 0x90 0x0 0x0 0x0 0x0 0x2 0x2 0x5 0x5 0x9 0x35 0xff 0xe8 0x0 0x2f 0x2 0x27 0x0 0xe5 0xa 0x44 0x7 0x8 0x1 0xf4 0x1 0xf4 0x0 0xf1 0x0 0xb8 0xc 0xc6 0xc 0xba 0x4 0xf 0x0 0xb2 0xe0 0x12 0x0 0xa 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x53 0x0 0x0 0x0 0x2 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xcc 0x58 0x0 0x0 0x0 0x0 0x4d 0xad 0x0 0x1c 0x0 0x0 0x48 0x6a 0x0 0x10 0x0 0x0 0x4b 0xd6 0x0 0x0 0x0 0x0 0x31 0xde 0x0 0x1f 0x0 0x0 0xcd 0x53 0x0 0xe 0x0 0x0 0x19 0xb1 0x0 0x1e 0x0 0x0 0x97 0x82 0x0 0xf 0x2 0xa1
Cleanup recv buffer before send: 0xf7 0x4 0xba 0x5 0x66 0x0 0x0 0x0 0x7 0x3 0xdd 0x0 0x0 0xff 0xfd 0x9 0x38 0x0 0x17 0x2 0x2f 0x9 0x6f 0x0 0x17 0x2 0x17 0xff 0xd9 0x2 0x38 0x13 0x8f 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xff 0xed 0x0 0x0 0x2 0x2b 0x1 0x1a 0x1 0x78 0x0 0x0 0x0 0x0 0x17 0x70 0xe8 0x90 0x0 0x0 0x0 0x0 0x2 0x2 0x5 0x5 0x9 0x37 0xff 0xe8 0x0 0x2f 0x2 0x27 0x0 0xe5 0xa 0x44 0x7 0x8 0x1 0xf4 0x1 0xf4 0x0 0xf1 0x0 0xb8 0xc 0xc6 0xc 0xba 0x4 0xf 0x0 0xb2 0xe0 0x15 0x0 0xa 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x53 0x0 0x0 0x0 0x2 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xcc 0x58 0x0 0x0 0x0 0x0 0x4d 0xad 0x0 0x1c 0x0 0x0 0x48 0x6a 0x0 0x10 0x0 0x0 0x4b 0xd6 0x0 0x0 0x0 0x0 0x31 0xde 0x0 0x1f 0x0 0x0 0xcd 0x53 0x0 0xe 0x0 0x0 0x19 0xb1 0x0 0x1e 0x0 0x0 0x97 0x82 0x0 0xf 0xd4 0x21

This error originated from a custom integration.

Logger: custom_components.foxess_modbus.modbus_controller
Source: custom_components/foxess_modbus/modbus_controller.py:153
Integration: FoxESS - Modbus (documentation, issues)
First occurred: 1:49:35 AM (1178 occurrences)
Last logged: 6:44:01 AM

General exception when polling /dev/ttyUSB0 247: ValueError('filedescriptor out of range in select()')
Traceback (most recent call last):
  File "/config/custom_components/foxess_modbus/modbus_controller.py", line 153, in _refresh
    reads = await self._client.read_registers(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/foxess_modbus/modbus_client.py", line 218, in read_registers
    response = await self._async_pymodbus_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/foxess_modbus/modbus_client.py", line 306, in _async_pymodbus_call
    result = await self._hass.async_add_executor_job(call, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/mixin.py", line 117, in read_input_registers
    pdu_reg_read.ReadInputRegistersRequest(address, count, slave, **kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/base.py", line 177, in execute
    self.delay_ms = self.params.reconnect_delay
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 168, in execute
    )
      
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 285, in _transact
    return size, None
       ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/transaction.py", line 325, in _send
    def _recv(self, expected_response_length, full):  # pylint: disable=too-complex
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/framer/rtu_framer.py", line 286, in sendPacket
    break
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/serial.py", line 231, in send
    self.last_frame_end = None
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/serial/serialposix.py", line 640, in write
    abort, ready, _ = select.select([self.pipe_abort_write_r], [self.fd], [], None)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: filedescriptor out of range in select()

home-assistant_2023-11-07T06-48-03.419Z.zip

commented

Are you sure you're running v1.8.0b3? The line numbers in that stack trace line up with v1.7.6, but not v1.8.0b3.

Remember you need to restart HA (not just a "quick reload") after updating/installing custom integrations.

commented
commented

You need to go to the github web interface to upload images

commented

Apologies... :)
modbus

commented

Aha, no, that's my mistake.

Slightly confused by what's going on in pyserial there. I'll have a deeper look when I get the chance

commented

Sorry for the delay!

Looks like this is a known issue in pyserial: pyserial/pyserial#617.

I've had a go at working around the issue. Can you give v1.8.0b5 a try please?

commented
commented

Thanks!

commented

Hi, any news on whether it's behaving better?

commented
commented

Thanks! It's likely a different problem to last time. I'm afraid I don't have a serial devie here, so I'm having to develop this workaround by proxy.

commented
commented

Thanks! Are you sure you're running 1.8.0b5? We shouldn't be using serialposix.py any more.

That said, my replacement for serialposix.py might still have an issue on that line - I'll look into it.

commented
commented

Right! I've had a chance to properly look at this, and it turns out that pyserial's attempt to use poll rather than select just addresses the read operation, and not the write operation.

So I've updated my workaround to also provide a write implementation which uses poll (completely untested again, as I don't have a serial device). Can you give v1.8.0b6 a try please?

commented
commented
commented

Thank you!

commented

Hi, any news?

commented
commented

Good to hear, thanks!

commented

Hi, is it still holding up a week in?

commented
commented

Fab, thank you! I'll get that released properly at some point.