drawing boundary from shapefile fails in recent conda builds

[Alan Iwi]

Recent builds of ncl 6.6.2 on conda-forge do not correctly display a boundary from a shapefile on a map when using the test code which is shown at cedadev/ceda-jaspy-envs#56. (That issue contains the code, the input data for the test, and the graphical output that is obtained in working and broken cases, so I won't repeat it all here.)

This is tested on Linux (CentOS 7 for x86_64), using a miniconda environment that was initially created using the latest available version as described here, and where earlier versions of ncl were installed for testing using for example conda install -y ncl=6.6.2=h8482486_9.

It seems that ncl=6.6.2=h8482486_9 is the last working release, and ncl=6.6.2=had3379d_10 is the first broken one. The above linked issue shows also some other release numbers that were tested.

[Rick Brownrigg]

Hi, Within the past week, there has been a report of a shapefile drawing correctly on one computer, but not on another seemingly similar machine using the same script and shapefile. The user tracked it down to the fact that the x/y variables in the shapefile appeared to be swapped on one machine versus the other. I strongly suspect that's what is going on here: ncl 1> load "./shapefile_utils.ncl" ncl 2> sname = "england_gor_2011.shp" ncl 3> print_shapefile_info(sname) (0) ====================================================================== (0) Filename: "england_gor_2011.shp" (0) Geometry type: polygon (0) # of features: 1 (0) Min/max lat: -0.51/ 0.33 (0) Min/max lon: 51.29/ 51.69 (0) Variable names and their types: (0) geometry : integer (0) segments : integer (0) x : double (0) y : double (0) label : string (0) name : string (0) code : string (0) ====================================================================== If, as implied by the filename, the shapefile covers a portion of england, then indeed those lat/lon values are swapped! Unfortunately, there is as of yet no explanation for the why/how this is happening. I suppose as a short-term work-around you could "manually" swap the variables in your script, but that definitely qualifies as a first-class hack :-/ Probably better to down-version your NCL installation. FWIW... Rick

…

On Sun, Jan 10, 2021 at 9:09 AM Alan Iwi ***@***.***> wrote: Recent builds of ncl 6.6.2 on conda-forge do not correctly display a boundary from a shapefile on a map when using the test code which is shown at cedadev/ceda-jaspy-envs#56 <cedadev/ceda-jaspy-envs#56>. (That issue contains the code, the input data for the test, and the graphical output that is obtained in working and broken cases, so I won't repeat it all here.) This is tested on Linux (CentOS 7 for x86_64), using a miniconda environment that was initially created using the latest available version as described here <https://www.ncl.ucar.edu/Download/conda.shtml>, and where earlier versions of ncl were installed for testing using for example conda install -y ncl=6.6.2=h8482486_9. It seems that ncl=6.6.2=h8482486_9 is the last working release, and ncl=6.6.2=had3379d_10 is the first broken one. The above linked issue shows also some other release numbers that were tested. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#151>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLWOXSWWBH3KAY7PMAKFM3SZHGMNANCNFSM4V4Q43BQ> .

[Alan Iwi]

@rbrownrigg Thank you for that. I will pass the information on to the user who reported the issue, although it is evident from what he wrote to me that it is the same person who has already been in contact with you.

My involvement with this is that I help to maintain software installations on a multi-user system, which is the machine on which it was not working. We are using conda environments to provide NCL and other software. After experimenting with installation of different releases of ncl from conda-forge, I now have both working and broken test cases on that same machine.

For example, with ncl=6.6.2=h297df27_19 (currently the default for users) I get the following from print_shapefile_info:

(0)	   Min/max lat:     -0.51/   0.33
(0)	   Min/max lon:     51.29/  51.69

while with ncl=6.6.2=h8482486_9 I get:

(0)	   Min/max lat:     51.29/  51.69
(0)	   Min/max lon:     -0.51/   0.33

The latter is also what the user gets on his other machine.

The user indicated to me that the boundary is meant to be London, so yes the shapefile is intended to cover England as the filename implies.

If there is a release that fixes the issue by the time that we build our next conda environment (which is hopefully soon), then we will upgrade to it. Otherwise we will downgrade.

Thanks for your help. I'm hoping that the information about the first broken version (working in h8482486_9, broken in had3379d_10) will help you to track down the source of the problem.

[Dave Allured]

@alanawi, I think this problem may be due to a version change in one of the underlying shapefile support libraries. Can you easily show library version numbers for h8482486_9 and had3379d_10?

[Dave Allured]

This problem is reproduced with a recent Macports build of ncarg/ncl 6.6.2.

[Dave Allured]

Perhaps it will not be necessary to look up the historical version info. NCL uses GDAL, which uses PROJ for coordinate transformations. Both libraries have recently seen major upgrades. In particular, PROJ has advanced from version 4 to version 7 since NCAR's standard NCL builds, about two years ago. There is a FAQ for PROJ that talks about ordering of coordinates:

Why is the axis ordering in PROJ not consistent?

And my favorite quote:

Unfortunately most GIS software on the market doesn’t follow this standard.

[Alan Iwi]

@Dave-Allured Thanks for your update (only just seen it - my username was misspelt when you tagged me, so it didn't email me).

Please can you tell me what the plan is to fix this problem, as I am a little confused about where we go from here (and wondering whether to downgrade now).

Re your question about library versions, I am not completely sure whether it still needs answering, but here's my best attempt to answer it.

I have downloaded the two tarballs:

• https://conda.anaconda.org/conda-forge/linux-64/ncl-6.6.2-h8482486_9.tar.bz2
• https://conda.anaconda.org/conda-forge/linux-64/ncl-6.6.2-had3379d_10.tar.bz2

to have a look. In terms of the libraries that they depend on, I can see (using ldd on the ncl executable), that:

• h8482486_9 depends on libgdal.so.20 and libproj.so.13
• had3379d_10 depends on libgdal.so.26 and libproj.so.15

In terms of what is provided by each of these packages, they don't seem to contain any *.so* files, and although the file contents differ, the list of paths is exactly the same.

Many thanks.

[Dave Allured]

@alaniwi, sorry about misspelling your user name.

what the plan is to fix this problem

Someone will need to fully investigate this bug and come up with a fix. Either NCAR will need to dedicate staff time, or else a community volunteer will be needed. This issue is in limbo until one of those two things happens. I am personally not able to dedicate time to this right now, but I try to add pointers when I think they might be relevant.

There is one thing that would help me a little. You gave so version numbers for gdal and proj, but can you also come up with their package version numbers for h8482486_9 and had3379d_10? Something like "gdal 2.4.2" and "proj 4.9.3"? This would help me get a better focus into the published release notes. I think you understand conda better than me.

[Dave Allured]

Please see issue #176 for a revised summary of this issue, and a possible workaround in the form of a modified shapefile_utils.ncl.

[Dave Allured]

Update. Here are some build details from conda-forge:

Build ID	Build date	GDAL version
h8482486_9	2019 Aug 31	libgdal >=2.4.1, <2.5.0a0
had3379d_10	2019 Sep 03	libgdal >=3.0.1, <3.1.0a0

There was a significant change in lat/lon ordering, starting in GDAL 3.0.0 (2019 May 5). The order now depends on details of the coordinate reference system (CRS) used to create the shapefile. Because of that, the order can now vary between shapefiles. I think this explains the original problem reported here by @alaniwi.

GDAL 3.x includes methods to get consistent ordering. NCL source code needs to be patched with one of these methods, to be able to correctly read all shapefiles. Refer to section MIGRATION GUIDE FROM GDAL 2.4 to GDAL 3.0 here:
https://github.com/OSGeo/gdal/blob/master/MIGRATION_GUIDE.TXT
Specifically, see the note about using SetAxisMappingStrategy.