reader.ept does not appear to be streaming
kjwaters opened this issue · comments
Describe the bug
Running a pipeline with EPT input has unexpectedly high memory usage.
pdal pipeline --stdin
With version 2.6.3, running pdal pipeline on the following pipeline uses 22 Gb of RAM, though both parts are streamable. It should be a low memory operation. The amount of memory used is in the ballpark of what would be needed to store the point cloud in memory, perhaps even more.
[
{
"type": "readers.ept",
"filename": "https://s3-us-west-2.amazonaws.com/usgs-lidar-public/SC_SavannahPeeDee_7_2019/ept.json",
"tag": "input_SC_SavannahPeeDee_7_2019",
"bounds": "([2300253.0, 2316273.0], [366099.0, 382119.0])/EPSG:2273"
},
{
"filename": "point_peedee7.las",
"type": "writers.las",
"forward": "all",
"minor_version":4
}
]Expected behavior
I expect this operation to use very little memory, essentially writing each point out as it is received. I do not expect it to scale with the size of the input.
System/installation information:
Please provide information on your PDAL version (pdal --version) and system (e.g., uname -a or ver).
$ uname -a
Linux ocm-s-fafnir 3.10.0-1160.108.1.el7.x86_64 #1 SMP Thu Jan 4 14:55:32 EST 2024 x86_64 x86_64 x86_64 GNU/Linux
pdal 2.6.3 (git-version: d12128)
If installed via Conda, you may be asked to paste the output of conda list and conda info as well.
$ conda list
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
annotated-types 0.6.0 pyhd8ed1ab_0 conda-forge
aws-c-auth 0.7.16 h79b3bcb_6 conda-forge
aws-c-cal 0.6.10 hb29e0c7_1 conda-forge
aws-c-common 0.9.13 hd590300_0 conda-forge
aws-c-compression 0.2.18 hecc5fa9_1 conda-forge
aws-c-event-stream 0.4.2 hf9b2f7b_4 conda-forge
aws-c-http 0.8.1 h5d7533a_5 conda-forge
aws-c-io 0.14.5 h50678d4_1 conda-forge
aws-c-mqtt 0.10.2 hf479d2b_4 conda-forge
aws-c-s3 0.5.2 h4ad9680_0 conda-forge
aws-c-sdkutils 0.1.15 hecc5fa9_1 conda-forge
aws-checksums 0.1.18 hecc5fa9_1 conda-forge
aws-crt-cpp 0.26.2 h19f5d62_7 conda-forge
aws-sdk-cpp 1.11.267 h5606698_1 conda-forge
azure-core-cpp 1.10.3 h91d86a7_1 conda-forge
azure-storage-blobs-cpp 12.10.0 h00ab1b0_0 conda-forge
azure-storage-common-cpp 12.5.0 hb858b4b_2 conda-forge
blosc 1.21.5 h0f2a231_0 conda-forge
boto3 1.34.59 pyhd8ed1ab_0 conda-forge
botocore 1.34.59 pyge38_1234567_1 conda-forge
brotli-python 1.1.0 py39h3d6467e_1 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.23.0 hd590300_0 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cairo 1.18.0 h3faef2a_0 conda-forge
ceres-solver 2.2.0 h2ea6fbf_1 conda-forge
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
cfitsio 4.3.1 hbdc6101_0 conda-forge
cftime 1.6.3 py39h44dd56e_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
curl 8.5.0 hca28451_0 conda-forge
draco 1.5.6 hf52228f_0 conda-forge
eigen 3.4.0 h00ab1b0_0 conda-forge
expat 2.6.1 h59595ed_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 h77eed37_1 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetds 1.4.10 h0535610_0 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
freexl 2.0.0 h743c826_0 conda-forge
fsspec 2024.2.0 pyhca7485f_0 conda-forge
gdal 3.8.4 py39he9866c7_1 conda-forge
geos 3.12.1 h59595ed_0 conda-forge
geotiff 1.7.1 h6b2125f_15 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h0b41bf4_3 conda-forge
glog 0.6.0 h6f12383_0 conda-forge
gmp 6.3.0 h59595ed_0 conda-forge
h5py 3.10.0 nompi_py39h2c511df_101 conda-forge
hdf4 4.2.15 h2a13503_7 conda-forge
hdf5 1.14.3 nompi_h4f84152_100 conda-forge
icu 73.2 h59595ed_0 conda-forge
idna 3.6 pyhd8ed1ab_0 conda-forge
importlib-metadata 7.0.0 pyha770c72_0 conda-forge
importlib_metadata 7.0.0 hd8ed1ab_0 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
jq 1.6 h27cfd23_1000
json-c 0.17 h7ab15ed_0 conda-forge
kealib 1.5.3 h2f55d51_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.21.2 h659d440_0 conda-forge
lcms2 2.16 hb7c19ff_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20230802.1 cxx17_h59595ed_0 conda-forge
libaec 1.1.2 h59595ed_1 conda-forge
libarchive 3.7.2 h2aa1ff5_1 conda-forge
libblas 3.9.0 20_linux64_openblas conda-forge
libboost-headers 1.83.0 ha770c72_0 conda-forge
libcblas 3.9.0 20_linux64_openblas conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libcurl 8.5.0 hca28451_0 conda-forge
libdeflate 1.19 hd590300_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libexpat 2.6.1 h59595ed_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h807b86a_3 conda-forge
libgdal 3.8.4 h13e47b0_1 conda-forge
libgfortran-ng 13.2.0 h69a702a_3 conda-forge
libgfortran5 13.2.0 ha4646dd_3 conda-forge
libglib 2.78.4 hf2295e7_4 conda-forge
libgomp 13.2.0 h807b86a_3 conda-forge
libgoogle-cloud 2.17.0 h884d01b_1 conda-forge
libgoogle-cloud-storage 2.17.0 hc7a4891_1 conda-forge
libgrpc 1.59.3 hd6c4280_0 conda-forge
libhwloc 2.9.2 default_h554bfaf_1009 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libjpeg-turbo 3.0.0 hd590300_1 conda-forge
libkml 1.3.0 h01aab08_1018 conda-forge
liblapack 3.9.0 20_linux64_openblas conda-forge
libnetcdf 4.9.2 nompi_h9612171_113 conda-forge
libnghttp2 1.58.0 h47da74e_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libopenblas 0.3.25 pthreads_h413a1c8_0 conda-forge
libpng 1.6.43 h2797004_0 conda-forge
libpq 16.2 h33b98f1_0 conda-forge
libprotobuf 4.24.4 hf27288f_0 conda-forge
libre2-11 2023.09.01 h7a70373_1 conda-forge
librttopo 1.1.0 h8917695_15 conda-forge
libspatialite 5.1.0 h7bd4643_4 conda-forge
libsqlite 3.45.1 h2797004_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_3 conda-forge
libtiff 4.6.0 ha9c0a0a_2 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.3.2 hd590300_0 conda-forge
libxcb 1.15 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libxml2 2.12.5 h232c23b_0 conda-forge
libzip 1.10.1 h2629f0a_3 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
lzo 2.10 h516909a_1000 conda-forge
markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge
mdurl 0.1.0 pyhd8ed1ab_0 conda-forge
meshio 5.3.4 pyhd8ed1ab_0 conda-forge
metis 5.1.0 h59595ed_1007 conda-forge
minizip 4.0.3 h0ab5242_0 conda-forge
mpfr 4.2.1 h9458935_0 conda-forge
ncurses 6.4 h59595ed_2 conda-forge
netcdf4 1.6.4 nompi_py39h4282601_103 conda-forge
nitro 2.7.dev8 h59595ed_0 conda-forge
nspr 4.35 h27087fc_0 conda-forge
nss 3.98 h1d7d5a4_0 conda-forge
numpy 1.26.2 py39h474f0d3_0 conda-forge
oniguruma 6.9.9 hd590300_0 conda-forge
openjpeg 2.5.2 h488ebb8_0 conda-forge
openssl 3.2.1 hd590300_0 conda-forge
packaging 24.0 pyhd8ed1ab_0 conda-forge
pathlib 1.0.1 py39hf3d152e_7 conda-forge
pcre2 10.43 hcad00b1_0 conda-forge
pdal 2.6.3 h312035a_2 conda-forge
pdal-plugins 1.3.0 pypi_0 pypi
pip 24.0 pyhd8ed1ab_0 conda-forge
pixman 0.42.2 h59595ed_0 conda-forge
planetary-computer 1.0.0 pyhd8ed1ab_0 conda-forge
poppler 24.02.0 h590f24d_0 conda-forge
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 16.2 h7387d8b_0 conda-forge
proj 9.3.1 h1d62c97_0 conda-forge
proj-data 1.17 hd8ed1ab_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pydantic 2.6.3 pyhd8ed1ab_0 conda-forge
pydantic-core 2.16.3 py39h9fdd4d6_0 conda-forge
pygments 2.17.2 pyhd8ed1ab_0 conda-forge
pymssql 2.2.8 py39h3d6467e_0 conda-forge
pyproj 3.6.1 py39h15b0fa6_5 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pystac 1.9.0 pyhd8ed1ab_0 conda-forge
pystac-client 0.7.6 pyhd8ed1ab_0 conda-forge
python 3.9.18 h0755675_1_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-dotenv 1.0.1 pyhd8ed1ab_0 conda-forge
python-pdal 3.3.0 py39hed70b7f_4 conda-forge
python_abi 3.9 4_cp39 conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
re2 2023.09.01 h7f4b329_1 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
rich 13.7.0 pyhd8ed1ab_0 conda-forge
s2n 1.4.5 h06160fa_0 conda-forge
s3fs 0.4.2 py_0 conda-forge
s3transfer 0.10.0 pyhd8ed1ab_0 conda-forge
setuptools 68.2.2 pyhd8ed1ab_0 conda-forge
shapely 2.0.3 py39h6404dd3_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
sqlite 3.45.1 h2c6b66d_0 conda-forge
suitesparse 5.10.1 h9e50725_1 conda-forge
tbb 2021.10.0 h00ab1b0_0 conda-forge
tiledb 2.20.1 hd75ad12_1 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
typing-extensions 4.8.0 hd8ed1ab_0 conda-forge
typing_extensions 4.8.0 pyha770c72_0 conda-forge
tzcode 2023c h0b41bf4_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
unixodbc 2.3.12 h661eb56_0 conda-forge
uriparser 0.9.7 hcb278e6_1 conda-forge
urllib3 1.26.18 pyhd8ed1ab_0 conda-forge
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.5 hac6953d_0 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.1.1 hd590300_0 conda-forge
xorg-libsm 1.2.4 h7391055_0 conda-forge
xorg-libx11 1.8.7 h8ee46fc_0 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxrender 0.9.11 hd590300_0 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.5 hfc55251_0 conda-forge
$ conda info
active environment : pdal_latest
active env location : /san1/tcm-i/miniconda3/envs/pdal_latest
shell level : 2
user config file : /csc/nac1/crs/kwaters/.condarc
populated config files : /csc/nac1/crs/kwaters/.condarc
conda version : 23.11.0
conda-build version : not installed
python version : 3.9.18.final.0
solver : libmamba (default)
virtual packages : __archspec=1=nehalem
__conda=23.11.0=0
__glibc=2.17=0
__linux=3.10.0=0
__unix=0=0
base environment : /san1/tcm-i/miniconda3 (writable)
conda av data dir : /san1/tcm-i/miniconda3/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /san1/tcm-i/miniconda3/pkgs
/csc/nac1/crs/kwaters/.conda/pkgs
envs directories : /san1/tcm-i/miniconda3/envs
/csc/nac1/crs/kwaters/.conda/envs
platform : linux-64
user-agent : conda/23.11.0 requests/2.31.0 CPython/3.9.18 Linux/3.10.0-1160.108.1.el7.x86_64 rhel/7.9 glibc/2.17 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.6
UID:GID : 1147:1001
netrc file : None
offline mode : False
Reproduced.