PDAL / PDAL

PDAL is Point Data Abstraction Library. GDAL for point cloud data.

Home Page:https://pdal.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

(pdal pipeline readers.copc Error) GDAL failure (1) PROJ: utm: Invalid latitude

kjwaters opened this issue · comments

Describe the bug
I'm running a pipeline with a COPC reader that includes a bounds definition in a different projection than the data. I'm getting an unexpected error reported of invalid latitudes, though I don't think it's actually losing any data.
The error is "(pdal pipeline readers.copc Error) GDAL failure (1) PROJ: utm: Invalid latitude" repeated many times.

$ pdal pipeline -i pipe.json
{
  "pipeline":[
    {
    "type": "readers.copc",
    "filename": "https://ocmgeodatastor1.blob.core.windows.net/lidar-copc/529/20070717_098821_w_ld_p31.copc.laz",
    "bounds": "([383868.0, 384691], [2950995.0, 2952001])/EPSG:3724"
    }
  ]
}

Expected behavior
The pipeline should not have given any errors or warnings.
If I don't specify the bounds in the reader, but instead add a filters.reprojection for EPSG:3724 and a filters.crop for the same bounds,
I don't see any errors and get the same number of points.
While I don't think this is causing any issues, I'm not positive and it's a bit disconcerting. All the points should be well within the UTM zone.

System/installation information:
Please provide information on your PDAL version (pdal --version) and system (e.g., uname -a or ver).

$ uname -a
Linux 260f4c13d9b3 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ pdal --version
pdal 2.7.0 (git-version: d5d146)

If installed via Conda, you may be asked to paste the output of conda list and conda info as well.

$ conda list
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aenum                     3.1.15                   pypi_0    pypi
archspec                  0.2.2              pyhd8ed1ab_0    conda-forge
aws-c-auth                0.7.16               h79b3bcb_6    conda-forge
aws-c-cal                 0.6.10               hb29e0c7_1    conda-forge
aws-c-common              0.9.13               hd590300_0    conda-forge
aws-c-compression         0.2.18               hecc5fa9_1    conda-forge
aws-c-event-stream        0.4.2                hf9b2f7b_4    conda-forge
aws-c-http                0.8.1                h5d7533a_5    conda-forge
aws-c-io                  0.14.5               h50678d4_1    conda-forge
aws-c-mqtt                0.10.2               hf479d2b_4    conda-forge
aws-c-s3                  0.5.2                h4ad9680_0    conda-forge
aws-c-sdkutils            0.1.15               hecc5fa9_1    conda-forge
aws-checksums             0.1.18               hecc5fa9_1    conda-forge
aws-crt-cpp               0.26.2               h19f5d62_7    conda-forge
aws-sdk-cpp               1.11.267             h5606698_1    conda-forge
azure-core-cpp            1.10.3               h91d86a7_1    conda-forge
azure-storage-blobs-cpp   12.10.0              h00ab1b0_0    conda-forge
azure-storage-common-cpp  12.5.0               hb858b4b_2    conda-forge
blosc                     1.21.5               h0f2a231_0    conda-forge
boltons                   23.1.1             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.24.0               hd590300_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
cairo                     1.18.0               h3faef2a_0    conda-forge
ceres-solver              2.2.0                h30ec75d_2    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310h2fee648_0    conda-forge
cfitsio                   4.4.0                hbdc6101_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.11.0         py310hff52083_1    conda-forge
conda-libmamba-solver     23.12.0            pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.0                    pypi_0    pypi
curl                      8.5.0                hca28451_0    conda-forge
cycler                    0.12.1                   pypi_0    pypi
dbf                       0.99.9                   pypi_0    pypi
debugpy                   1.8.1                    pypi_0    pypi
distro                    1.8.0              pyhd8ed1ab_0    conda-forge
draco                     1.5.7                h00ab1b0_0    conda-forge
eigen                     3.4.0                h00ab1b0_0    conda-forge
expat                     2.6.2                h59595ed_0    conda-forge
fmt                       10.1.1               h00ab1b0_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.50.0                   pypi_0    pypi
freetype                  2.12.1               h267a509_2    conda-forge
freexl                    2.0.0                h743c826_0    conda-forge
gdal                      3.8.4           py310he073c5f_3    conda-forge
geos                      3.12.1               h59595ed_0    conda-forge
geotiff                   1.7.1               h6b2125f_15    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
glog                      0.7.0                hed5481d_0    conda-forge
gmp                       6.3.0                h59595ed_1    conda-forge
hdf4                      4.2.15               h2a13503_7    conda-forge
hdf5                      1.14.3          nompi_h4f84152_100    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
json-c                    0.17                 h7ab15ed_0    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py310hff52083_3    conda-forge
kealib                    1.5.3                h2f55d51_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.5                    pypi_0    pypi
krb5                      1.21.2               h659d440_0    conda-forge
laspy                     2.5.3                    pypi_0    pypi
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
libaec                    1.1.3                h59595ed_0    conda-forge
libarchive                3.7.2                h2aa1ff5_1    conda-forge
libblas                   3.9.0           21_linux64_openblas    conda-forge
libboost-headers          1.84.0               ha770c72_1    conda-forge
libcblas                  3.9.0           21_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.5.0                hca28451_0    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgdal                   3.8.4                hab4ef92_3    conda-forge
libgfortran-ng            13.2.0               h69a702a_5    conda-forge
libgfortran5              13.2.0               ha4646dd_5    conda-forge
libglib                   2.80.0               hf2295e7_1    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libgoogle-cloud           2.17.0               h884d01b_1    conda-forge
libgoogle-cloud-storage   2.17.0               hc7a4891_1    conda-forge
libgrpc                   1.59.3               hd6c4280_0    conda-forge
libhwloc                  2.9.3           default_h554bfaf_1009    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
libkml                    1.3.0             h01aab08_1018    conda-forge
liblapack                 3.9.0           21_linux64_openblas    conda-forge
libmamba                  1.5.5                had39da4_0    conda-forge
libmambapy                1.5.5           py310h39ff949_0    conda-forge
libnetcdf                 4.9.2           nompi_h9612171_113    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libpq                     16.2                 h33b98f1_1    conda-forge
libprotobuf               4.24.4               hf27288f_0    conda-forge
libre2-11                 2023.09.01           h7a70373_1    conda-forge
librttopo                 1.1.0               h8917695_15    conda-forge
libsolv                   0.7.27               hfc55251_0    conda-forge
libspatialite             5.1.0                h7bd4643_4    conda-forge
libsqlite                 3.45.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libtiff                   4.6.0                ha9c0a0a_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxml2                   2.12.6               h232c23b_0    conda-forge
libzip                    1.10.1               h2629f0a_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.5.5           py310h51d5547_0    conda-forge
matplotlib                3.8.2                    pypi_0    pypi
menuinst                  2.0.1           py310hff52083_0    conda-forge
metis                     5.1.0             h59595ed_1007    conda-forge
minizip                   4.0.5                h0ab5242_0    conda-forge
mpfr                      4.2.1                h9458935_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
nitro                     2.7.dev8             h59595ed_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.98                 h1d7d5a4_0    conda-forge
numpy                     1.26.4          py310hb13e2d6_0    conda-forge
openjpeg                  2.5.2                h488ebb8_0    conda-forge
openssl                   3.2.1                hd590300_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pcre2                     10.43                hcad00b1_0    conda-forge
pdal                      2.7.0                h312035a_0    conda-forge
pdal-plugins              1.3.0                    pypi_0    pypi
pillow                    10.2.0                   pypi_0    pypi
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
pixman                    0.43.2               h59595ed_0    conda-forge
platformdirs              4.1.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
poppler                   24.03.0              h590f24d_0    conda-forge
poppler-data              0.4.12               hd8ed1ab_0    conda-forge
postgresql                16.2                 h82ecc9d_1    conda-forge
proj                      9.3.1                h1d62c97_0    conda-forge
proj-data                 1.17                 hd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py310h2372a71_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyodbc                    5.1.0                    pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pyproj                    3.6.0                    pypi_0    pypi
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         hd12c33a_0_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
python-dotenv             1.0.0                    pypi_0    pypi
python-pdal               3.3.0           py310he6b3783_6    conda-forge
python_abi                3.10                    4_cp310    conda-forge
rclone                    1.65.2               h519d9b9_0    conda-forge
re2                       2023.09.01           h7f4b329_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.4.post0         hd590300_1    conda-forge
reproc-cpp                14.2.4.post0         h59595ed_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.18.5          py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.7           py310h2372a71_2    conda-forge
s2n                       1.4.5                h06160fa_0    conda-forge
saxonche                  12.4.2                   pypi_0    pypi
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   2.0.1                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
snappy                    1.1.10               h9fff704_0    conda-forge
sqlite                    3.45.2               h2c6b66d_0    conda-forge
suitesparse               5.10.1               h5a4f163_3    conda-forge
tbb                       2021.11.0            h00ab1b0_1    conda-forge
tiledb                    2.20.1               hd75ad12_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
tzcode                    2024a                h3f72095_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
uriparser                 0.9.7                h59595ed_1    conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.5                hac6953d_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.7                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml-cpp                  0.8.0                h59595ed_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstandard                 0.22.0          py310h1275a96_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge
$ conda info

I don't have an answer, just posting some debugging info and ideas.

When I run the pipeline with PROJ_DEBUG=3 set, I get the following:

(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: pj_ellipsoid - final:    ellps=GRS80
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: pj_ellipsoid - final:
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: Building arg list for step no. 0
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: init - proj=axisswap, 2
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline:     order=2,1
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: axisswap: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: axisswap: pj_ellipsoid - final:    ellps=GRS80
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: Step 0 (proj=axisswap) at 0x1122b5c80
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline at [0x1122b6080]:    step at [0x1122b5c80] (proj=axisswap) done
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: Building arg list for step no. 1
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: init - proj=unitconvert, 3
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline:     xy_in=deg
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline:     xy_out=rad
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: unitconvert: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: unitconvert: pj_ellipsoid - final:    ellps=GRS80
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: unitconvert: xy_in unit: Degree
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: unitconvert: xy_out unit: Radian
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: Step 1 (proj=unitconvert) at 0x1122b5880
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline at [0x1122b6080]:    step at [0x1122b5880] (proj=unitconvert) done
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: Building arg list for step no. 2
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: init - proj=utm, 3
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline:     zone=17
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline:     ellps=GRS80
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: utm: pj_ellipsoid - final: a=6378137.000 f=1/298.257, errno=0
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: utm: pj_ellipsoid - final:    ellps=GRS80
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: Step 2 (proj=utm) at 0x1122b5480
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline at [0x1122b6080]:    step at [0x1122b5480] (proj=utm) done
(pdal pipeline readers.copc Debug) GDAL debug: PROJ_TRACE: pipeline: Pipeline: 3 steps built. Determining i/o characteristics

COPC of curvilinear coordinate systems, like this file has, is not such a great idea. Your octree ends up looking awful. See @connormanning's moonlidar.com talk for some consequences.

(pdal pipeline readers.copc Debug) 18 overlapping nodes

image

IMO it would be better to write the data into a rectilinear 3D coordinate system with the units all being the same. Some kind of ECEF like was done with moonlidar.com is one way to do it.

ECEF coordinate systems aren't so friendly for people for other reasons though. There's no silver bullet.