Unstable camera connection handshakes

[kesaroid]

Describe what you want to implement and what the issue & the steps to reproduce it are:

I am trying to run 28 cameras with sufficient network bandwidth on Ubuntu 22.04
The pylon viewer 7.4.0 seems to be buggy and just disappears and reappears cameras randomly.
But if the continuous shot is open, even if the camera disappears from devices, we still see the live feed.

While trying to do it with code using pypylon, it is able to open a few cameras and randomly crashes. The number of cameras that open are random.

Pylon viewer randomly detects and removes the cameras. But I am constantly able to ping the cameras.
Once the handshake is complete and I have initiated the cameras, even if they disappear from pylon-viewer saying - camera removed, we are still able to read the frames through pypylon.

Is your camera operational in Basler pylon viewer on your platform

Yes

Hardware setup & camera model(s) used

CPU architecture: X86_64
Operating System: Ubuntu 22.04 Update 3
RAM: 128gb

Interfaces used to connect the cameras:
NIC: Broadcom 57504 quad NIC 10GBE
Switches: 2x Netgear M4300-28G-POE+
Cable types/lengths: Combination of CAT6a, fiber/25 meters

Runtime information:

python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
platform: linux/x86_64/5.15.0-102-generic
pypylon: 3.0.1 / 7.4.0.38864

[SMA2016a]

just for my information: is there is good reason why you do not use a 52port switch?
<The pylon viewer 7.4.0 seems to be buggy> while running image acquisition or just keep the cameras open?

what is the camera model that you are using?

what is the subnet mask of both IP ranges?

have you optimized your system using pylon GigE configurator?

[kesaroid]

The primary reason was bandwidth issues for the uplink ports. We have divided the cameras evenly between the 2 switches, under the same subnet mask 255.255.255.0
All of the camera models are the same: a2A1920-51gcPRO

Upon using the pylon GigE configurator, I get

[2024-04-22 15:34:18] - Information: ******************************************************************
[2024-04-22 15:34:18] - Information: *  PylonGigEConfigurator 1.0.0.0 - Copyright (c) 2022 Basler AG  *
[2024-04-22 15:34:18] - Information: ******************************************************************
[2024-04-22 15:34:18] - Information: Optimizes system and network settings for Basler GigE cameras.
[2024-04-22 15:34:18] - Information: Note: Before using this tool, make sure only cameras are connected to your network adapter(s).
[2024-04-22 15:34:18] - Information: 
[2024-04-22 15:34:18] - Information: 
[2024-04-22 15:34:18] - Information: Auto configuration is looking for network adapters and cameras ...
[2024-04-22 15:34:18] - Information: Adapter "eno12399np0" (192.160.1.101) (BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (NetXtreme-E BCM57504 4x25G OCP3.0)) is faster than 1 Gbps but non-Basler. Excluding adapter from optimization.
[2024-04-22 15:34:18] - Information: Adapter "eno12409np1" (192.160.2.102) (BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (NetXtreme-E BCM57504 4x25G OCP3.0)) is faster than 1 Gbps but non-Basler. Excluding adapter from optimization.
[2024-04-22 15:34:26] - Information: Adapter "eno12419np2" (192.168.3.2) (BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (NetXtreme-E BCM57504 4x25G OCP3.0)) has no camera attached. Excluding adapter from optimization.
[2024-04-22 15:34:26] - Information: Adapter "eno12429np3" (192.168.4.2) (BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (NetXtreme-E BCM57504 4x25G OCP3.0)) has no camera attached. Excluding adapter from optimization.
[2024-04-22 15:34:26] - Information: Adapter "eno8303" (10.40.49.24) (NetXtreme BCM5720 Gigabit Ethernet PCIe) has no camera attached. Excluding adapter from optimization.
[2024-04-22 15:34:26] - Information: Adapter "eno8403" (192.168.5.2) (NetXtreme BCM5720 Gigabit Ethernet PCIe) has no camera attached. Excluding adapter from optimization.
[2024-04-22 15:34:26] - Information: No network adapter with an attached camera found. Exiting.

eno12399np0 & eno12409np1 are the adapters that have cameras connected to it. It says non-basler for some reason. Is it possible that pylon GigE configurator only supports specific NIC?

This is how the cameras are configured

However cameras are removed and connected at random (especially the ones connected to eno12409np1)
Upon running bandwidth manager with the cameras connected to eno12409np1, usually pylon crashes and gives the following error

QThreadPipe: Unable to create pipe: Too many open files
[2024-04-22 11:45:53.824][Fatal] default: QEventDispatcherUNIXPrivate(): Can not continue without a thread pipe
Aborted (core dumped)

We are unable to pinpoint if this is a hardware, network configuration or pylon-viewer issue.

[SMA2016a]

You need to increase the number of file descriptor if you have multiple cameras.

ulimit -sn 4096

[kesaroid]

Thank you @SMA2016a Increasing the file descriptor helped make pylon-viewer more stable.
I understand that there were a lot of issues that I was trying to comprise in this issue, and the community was able to help us.
We were able to fix the issue by switching out the switch ports for the unstable cameras.