(Bug): Natron render random crashes on farm reading tiff images

Question

(Bug): Natron render random crashes on farm reading tiff images

lpequignot opened this issue 2 years ago · comments

lpequignot commented 2 years ago

Make sure to follow our issue report guidelines

I'm using the latest version of Natron (not required but recommended)
I've restarted Natron and the issue persists
I've run Natron via the command line and the issue persists
I've followed the contributing guidelines to the best of my understanding
My issue is not on the issue tracker or in a pull request already (go search for it and dig around a little bit!)
This bug is reproducible

Natron version

Natron 2.4.3

Operating system

Rocky linux release 8.6

System specs

RAM: 250G
CPU: AMD EPYC 75F3 32-Core Processor
GPU: Integrated Matrox G200eW3 Graphics Controller

Did you install Natron using the official installer?

Yes, I used the official installer
No, I installed from a binary archive
No, I compiled Natron from sources
No, I installed Natron via another method

Custom installation path

/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer

What were you trying to do?

Rendering a ntp file on farm using Deadline

Full Command: "/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer" -w Write_EXR 4701-4800 "/var/lib/Thinkbox/Deadline10/workers/rf076/jobsData/63d05790f08ce30d267e43ec/thread0_temp38RQA0/m2m_20220329_apollo_houston_io_lowpano_8972_v001.ntp"

What did you expect to happen? What happened instead?

We had random crashes, always the same traceback (provided in additional details).
We found that it was because we were using tiff file format (for STMaps).
After using exr files instead of tiff, it solved the issue.

It seems that it's a threading issue that comes form the libtiff itself used in OpenImageIO.

Maybe, an update of Oiio could fix the issue as in the latest changes there are fixes concerning libtiff threading issues.

https://github.com/OpenImageIO/oiio/blob/master/CHANGES.md

TIFF:
Guard against corrupt files with buffer overflows. (TALOS-2022-1627, CVE-2022-41977) #3628 (2.4.5/2.5.0.0)
Guard against buffer overflow for certain CMYK files. (TALOS-2022-1633, CVE-2022-41639) (TALOS-2022-1643, CVE-2022-41988) #3632 (2.4.5/2.5.0.0)
While building against the new libtiff 4.5, use its new per-tiff error handlers to ensure better thread safety. #3719 (2.5.0.0/2.4.8.0)

Step-by-step reproduction instructions

rendering the ntp script on the farm

Additional details

2023-01-24 17:17:52: 0: STDOUT: Loading project: /var/lib/Thinkbox/Deadline10/workers/rf076/jobsData/63d05790f08ce30d267e43ec/thread0_temp38RQA0/m2m_20220329_apollo_houston_io_lowpano_8972_v001.ntp
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc0453c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc0453c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc119e80] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc119e80] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc6aca80] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc6aca80] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xca1f480] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xca1f480] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xcd75680] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xcd75680] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd312600] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd312600] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd667980] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd667980] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd9ba2c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd9ba2c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xdf73b00] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xdf73b00] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xe2d8d40] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xe2d8d40] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xe63c180] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xe63c180] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xebd8000] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xebd8000] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xef33dc0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xef33dc0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xf27b7c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xf27b7c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xf8240c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xf8240c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xfb88840] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xfb88840] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xfebd9c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xfebd9c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1046bc40] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1046bc40] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x107b0480] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x107b0480] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x10af1600] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x10af1600] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x110a6040] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x110a6040] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1140a440] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1140a440] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11760ac0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11760ac0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11d048c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11d048c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1205c7c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1205c7c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1239ccc0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1239ccc0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x12952200] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x12952200] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: Write_EXR ==> Rendering started
2023-01-24 17:17:56: 0: STDOUT: Write_EXR ==> Frame: 4701, Progress: 1.0%, 0.3 Fps, Time Remaining: 5 minutes
2023-01-24 17:17:58: 0: STDOUT: Write_EXR ==> Frame: 4703, Progress: 2.0%, 0.4 Fps, Time Remaining: 4 minutes
2023-01-24 17:17:58: 0: STDOUT: Write_EXR ==> Frame: 4702, Progress: 3.0%, 0.5 Fps, Time Remaining: 2 minutes 56 seconds
2023-01-24 17:18:00: 0: STDOUT: Write_EXR ==> Frame: 4704, Progress: 4.0%, 0.5 Fps, Time Remaining: 2 minutes 56 seconds
2023-01-24 17:18:00: 0: STDOUT: Write_EXR ==> Frame: 4705, Progress: 5.0%, 0.7 Fps, Time Remaining: 2 minutes 21 seconds
2023-01-24 17:18:02: 0: STDOUT: Write_EXR ==> Frame: 4706, Progress: 6.0%, 0.6 Fps, Time Remaining: 2 minutes 28 seconds
2023-01-24 17:18:02: 0: STDOUT: Write_EXR ==> Frame: 4707, Progress: 7.0%, 0.7 Fps, Time Remaining: 2 minutes 5 seconds
2023-01-24 17:18:04: 0: STDOUT: Write_EXR ==> Frame: 4708, Progress: 8.0%, 0.7 Fps, Time Remaining: 2 minutes 11 seconds
2023-01-24 17:18:04: 0: STDOUT: Write_EXR ==> Frame: 4709, Progress: 9.0%, 0.8 Fps, Time Remaining: 1 minute 56 seconds
2023-01-24 17:18:06: 0: STDOUT: Write_EXR ==> Frame: 4710, Progress: 10.0%, 0.7 Fps, Time Remaining: 2 minutes 0 second
2023-01-24 17:18:06: 0: STDOUT: Write_EXR ==> Frame: 4711, Progress: 11.0%, 0.8 Fps, Time Remaining: 1 minute 48 seconds
2023-01-24 17:18:09: 0: STDOUT: Write_EXR ==> Frame: 4713, Progress: 12.0%, 0.7 Fps, Time Remaining: 1 minute 57 seconds
2023-01-24 17:18:09: 0: STDOUT: Write_EXR ==> Frame: 4712, Progress: 13.0%, 0.8 Fps, Time Remaining: 1 minute 48 seconds
2023-01-24 17:18:11: 0: STDOUT: Write_EXR ==> Frame: 4715, Progress: 14.0%, 0.8 Fps, Time Remaining: 1 minute 53 seconds
2023-01-24 17:18:11: 0: STDOUT: Write_EXR ==> Frame: 4714, Progress: 15.0%, 0.8 Fps, Time Remaining: 1 minute 45 seconds
2023-01-24 17:18:13: 0: STDOUT: Write_EXR ==> Frame: 4716, Progress: 16.0%, 0.8 Fps, Time Remaining: 1 minute 47 seconds
2023-01-24 17:18:13: 0: STDOUT: Caught segmentation fault (SIGSEGV) from thread Parallel render thread (0x7fc098004830)(0x7fc098004820), faulty address is 0x7fc1b81fec1f from 0x7fb89f227000
2023-01-24 17:18:13: 0: STDOUT: Backtrace:
2023-01-24 17:18:13: 0: STDOUT: [Frame 1]: /lib64/libc.so.6(+0xcfc1f) [0x7fc1b81fec1f]
2023-01-24 17:18:13: 0: STDOUT: [Frame 2]: /lib64/libc.so.6(+0xcfc1f) [0x7fc1b81fec1f]
2023-01-24 17:18:13: 0: STDOUT: [Frame 3]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/../../Libraries/libOpenImageIO.so.2.3(_ZN16OpenImageIO_v2_39TIFFInput8readspecEb+0xd62) [0x7fc1a374b002]
2023-01-24 17:18:13: 0: STDOUT: [Frame 4]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/../../Libraries/libOpenImageIO.so.2.3(_ZN16OpenImageIO_v2_39TIFFInput13seek_subimageEii+0x8e) [0x7fc1a374c78e]
2023-01-24 17:18:13: 0: STDOUT: [Frame 5]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/../../Libraries/libOpenImageIO.so.2.3(_ZN16OpenImageIO_v2_39TIFFInput4openERKSsRNS_9ImageSpecE+0x37) [0x7fc1a374cf57]
2023-01-24 17:18:13: 0: STDOUT: [Frame 6]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/../../Libraries/libOpenImageIO.so.2.3(_ZN16OpenImageIO_v2_310ImageInput4openERKSsPKNS_9ImageSpecEPNS_10Filesystem7IOProxyE+0x2cc) [0x7fc1a35747cc]
2023-01-24 17:18:13: 0: STDOUT: [Frame 7]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/IO.ofx(+0x15937a) [0x7fc1a5db037a]
2023-01-24 17:18:13: 0: STDOUT: [Frame 8]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/IO.ofx(+0x15a4ae) [0x7fc1a5db14ae]
2023-01-24 17:18:13: 0: STDOUT: [Frame 9]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/IO.ofx(+0x81535) [0x7fc1a5cd8535]
2023-01-24 17:18:13: 0: STDOUT: [Frame 10]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/IO.ofx(+0x1c99f3) [0x7fc1a5e209f3]
2023-01-24 17:18:13: 0: STDOUT: [Frame 11]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(ZN3OFX4Host11ImageEffect8Instance9mainEntryEPKcPKvPNS0_8Property3SetES9+0x115) [0xd44b35]
2023-01-24 17:18:13: 0: STDOUT: [Frame 12]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(ZN6Natron22OfxImageEffectInstance9mainEntryEPKcPKvPN3OFX4Host8Property3SetES9+0x48) [0x984148]
2023-01-24 17:18:13: 0: STDOUT: [Frame 13]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(_ZN3OFX4Host11ImageEffect8Instance27getRegionOfDefinitionActionEd9OfxPointDiR8OfxRectD+0x1d6) [0xd4be76]
2023-01-24 17:18:13: 0: STDOUT: [Frame 14]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(_ZN6Natron17OfxEffectInstance21getRegionOfDefinitionEmdRKNS_11RenderScaleENS_7ViewIdxEPNS_5RectDE+0x15d) [0x96d27d]
2023-01-24 17:18:13: 0: STDOUT: [Frame 15]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(_ZN6Natron8ReadNode21getRegionOfDefinitionEmdRKNS_11RenderScaleENS_7ViewIdxEPNS_5RectDE+0x91) [0xa3ad11]

Ole-André · Answer 1 · Sun Jan 29 2023 02:26:11 GMT+0800 (China Standard Time)

Thanks for the report.

Natron 2.5.0 on Windows and Mac includes new enough OIIO (2.4.5.0+ has tiff fixes) if i'm not mistaken, unsure about Linux. If the Linux version is off then we will update it before the next release.

Frédéric Devernay · Answer 2 · Sun Jan 29 2023 09:03:21 GMT+0800 (China Standard Time)

the threading-related libtiff fixes are for libtiff 4.5, and are only included in the upcoming oiio 2.5, due for summer 2023. We will upgrade both when oiio 2.5 is released

lpequignot · Answer 3 · Mon Jan 30 2023 22:14:03 GMT+0800 (China Standard Time)

Hello,

Thanks for the follow-up, we'll test the new version with oiio 2.5 when it'll be released.
In the meantime, we'll use another image format.