Save as pcd fails for clouds larger than 2GB

[Karl]

Describe the bug

Saving a pointcloud as pcd crashes when the size is above ~2GB on disk, this is not due to RAM limitations.

Error message when using the cmd line interface for export:
/usr/include/c++/13.2.0/bits/stl_vector.h:1125: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = unsigned char; _Alloc = std::allocator<unsigned char>; reference = unsigned char&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.

How to reproduce

1. Open a pointcloud of any format (larger than 2GB) (pcd and bin tested)
2. Save point cloud as Point Cloud Library file (.pcd)
3. See cloudcompare crash

Expected behaviour

Pointcloud being saved to disk at requested location as a .pcd with all fields intact

Additional context

Saving as cloudcompare bin works fine so seems to be an issue with how the pcd export function is implemented.

Your environment

• CC Version (or git hash/tag): 2.13.1 (Kharkiv - Apr 13 2024) [Linux 64-bit] as provided by flatpak
• OS & Version: Ubuntu 20.04.6 LTS
• Graphics card: Nvidia RTX 3070 8GB
• Qt Version (if compiling):

[Daniel Girardeau-Montaut]

Maybe it's also because of a lack of memory? Because the CloudCompare cloud will have to be converted to a PCL cloud first (at least doubling the required memory).

Else it's a limitation from PCL (which is in charge of creating the PCD file). But I would lean towards the first explanation. Can you monitor the memory usage?

[Karl]

Absolutely, I've done that previously. The full RAM is 64 GB and 48GB+ is available. The memory does increase a bit, with about the same size as the pointcloud but there's plenty left for it so I've ruled this out as a likely cause. I've also noticed the exact same behaviour on a MAC laptop.

PCL can normally handle larger pointclouds so it's probably not a limitation with the library itself..

[Daniel Girardeau-Montaut]

So there are only 3 steps in the PCD I/O filter 'saveFile' function:

• optional transformation of the coordinates to preserve the raw coordinates (see your other issue) --> this can be skipped if you remove the sensor associated to the cloud. According to your reply, you should have plenty of memory anyway.
• conversion of the CC cloud to a PCD cloud --> If this was failing on CC's side, this should lead to an error message like "the third-party library in charge of saving/loading the file has failed to perform the operation". And it seems that all the calls to the PCL library at this point are within 'try/catch' statements, so I would assume that if an error was raised by PCL during this process, it would probably be caught and reported with the same error message as above.
• saving the cloud by calling pcl::io::savePCDFile from the PLC library --> I see that this call was not inside a try/catch statement

I've added a 'try/catch' statement on this branch: https://github.com/CloudCompare/CloudCompare/tree/version_2.13.X

By any chance, are you able to test this change and see if the error message differs?

[tmontaigu]

The problem is still going to happen as it looks like its an assert that is hit and not an exception

On a test I did it seems like something goes wrong in pcl::concatenateFields

I suspect some kind of overflow happening:
Some variable is an int, its increased by some amount, and overflows, becomes negative, is then used to index into a vector, the variable is then casted to std::size_t, it becomes some very large index which are out of bounds and trigger the error.

I'll try to check with a local install of PCL, but https://github.com/PointCloudLibrary/pcl/blob/c6bbf02a084a39a02d9e2fc318a59fe2f1ff55c1/common/src/io.cpp#L178 could be the source

[tmontaigu]

I can confirm that there is a problem in PCL's function

[Karl]

Thanks for checking,
The binary pcd format is relatively straight forward and wouldn't need an actual conversion from CC using PCL, would there be of interest to have a stand-alone read/write method for pcd files in CC without the PCL dependency?

[tmontaigu]

Yes that could be interesting because the PCD writer badically duplicates the cloud in memory when converting it to PCD, so it would help avoid that inefficiency

But then, If you write your own PCD reader/writer you have to make sure its correct, and that it supports all features (e.g ASCII/BINARY/BINRAY_COMPRESSED)