GigE cameras periodically failing and/or displaying "Camera physically removed"

[matkir]

Describe what you want to implement and what the issue & the steps to reproduce it are:

We have multiple machines running, using the Basler Ace.
We have several GIGe Basler Ace cameras GigE connected to an linux machine, with dedicated gigabit PoE to each camera.

The cameras are constantly running, capturing data (as we see in code snippet 1).

code snippet 1

while True: 
    grab_result = camera.RetrieveResult(self.timeout, pylon.TimeoutHandling_ThrowException)
    if grab_result.GrabSucceeded():
        doSomethingWithTheResult(grab_result)
    else:
        print("Error grabbing")
    grab_result.Release()

Occasionally we read out extra data from the cameras (as we see in code snippet 2).

code snippet 2

var1 = camera.StreamGrabber.Statistic_Failed_Buffer_Count.GetValue()
var2 = camera.ExposureTimeAbs.GetValue()
var3 = camera.GainRaw.GetValue()

This code runs fine for long periods of time. but occasionally we get a burst of errors, only one camera at the time, and the camera varies between days.
When we get an error, we get multiple fails from reading the image, and all reading of parameters gives us "camera physically removed"

code snippet 1 gives us the following errors:

_genicam.TimeoutException: Grab timed out. Possible reasons are: The image transport from the camera device is not working properly, e.g., all GigE network packets for streaming are dropped; The camera uses explicit triggering (see TriggerSelector for more information) and has not been triggered; Single frame acquisition mode is used and one frame has already been acquired; The acquisition has not been started or has been stopped.

code snippet 2 gives us the following errors:

_genicam.RuntimeException: The camera device has been physically removed. : RuntimeException thrown (file 'InstantCameraImpl.h', line 2071)

As mentioned, this only happens very rarely, and only a few of the machines has this error.
We are grateful for any insight from the Basler side.
Our main goal is to keep the framerate as high as possible.

Is your camera operational in Basler pylon viewer on your platform

Yes

Hardware setup & camera model(s) used

Nvidia Jestson Xavier, Ubuntu 18, 32G ram.

Runtime information:

python: 3.6.9 (default, Mar 10 2023, 16:46:00) 
[GCC 8.4.0]
platform: linux/aarch64/4.9.140-l4t-r32.4
pypylon: 2.3.0 / 7.2.1.19268

[SMA2016a]

I don't think that this issue is related to pypylon. So I would kindly ask you to turn to your local distributor to get the technical support for doing the standard GigE troubleshooting.

do you same issue of you would remove your line doSomethingWithTheResult(grab_result)?

[matkir]

Yes, i can confirm that the issue persists regardless of "doSomethingWithTheResult".
we have also tried to remove most other irrelevant other code, we have tried to strip down the code to just the camera capture specific code.

Tank you for your answer, ill update the issue if it turns out its a pypylon specific problem

[kesaroid]

Hi @matkir do you still face the same issue? What is the total number of cameras you have?
Also, do you also see "camera physically removed" in pylon viewer?

[matkir]

We're still having issues.
We run up to 4 cameras machine. And as i mentioned, the errors happens randomly.

We have learned that different versions of pylon and pypylon have different error rates.
Right now, our most stable build is with pylon_7.4.0.14900 and pypylon 2.3.0