Failed Import - SRTM elevation - There was an issue looking up the coordinates for - Could not parse OSM file
torrswanson opened this issue · comments
Describe the bug
- Import fails at a very specific coordinate with SRTM elevation enabled. Can reproduce every time I run the import as described below.
To Reproduce
Steps to reproduce the behavior. For examples:
- checkout recent and stable version of GraphHopper
using graphhopper-web-8-0.jar
wget https://download.geofabrik.de/north-america/canada-latest.osm.pbf
java -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -Xmx100g -Xms100g -jar graphhopper-web-*.jar import config-example.yml
2024-01-22 23:07:32.948 [main] INFO c.g.routing.lm.LMPreparationHandler - Loaded built-in landmark splitting collection from jar:file:/opt/graphhopper/production/graphhopper-web-8.0.jar!/com/graphhopper/routing/lm/map.geo.json
2024-01-22 23:07:32.971 [main] INFO com.graphhopper.GraphHopper - version 8.0|2023-10-17T10:01:00Z (9,21,6,5,2,9)
2024-01-22 23:07:33.353 [main] INFO com.graphhopper.GraphHopper - No custom areas are used, custom_areas.directory not given
2024-01-22 23:07:33.362 [main] INFO com.graphhopper.GraphHopper - No country rules available
2024-01-22 23:07:33.362 [main] INFO com.graphhopper.GraphHopper - start creating graph from canada-latest.osm.pbf
2024-01-22 23:07:33.382 [main] INFO com.graphhopper.GraphHopper - using car,bike,foot|RAM_STORE|3D|turn_cost|nodes:9,edges:21,geometry:6,location_index:5,string_index:2,nodesCH:0,shortcuts:9, memory:totalMB:102400, usedMB:210
2024-01-22 23:07:33.399 [main] INFO c.g.reader.osm.WaySegmentParser - Start reading OSM file: 'canada-latest.osm.pbf'
2024-01-22 23:07:33.399 [main] INFO c.g.reader.osm.WaySegmentParser - pass1 - start
2024-01-22 23:08:31.826 [main] INFO c.g.reader.osm.WaySegmentParser - pass1 - start reading OSM ways
2024-01-22 23:08:46.533 [main] INFO c.g.reader.osm.WaySegmentParser - pass1 - processed ways: 10,000,000, accepted ways: 1,502,988, way nodes: 14,673,486, totalMB:102400, usedMB:22964
2024-01-22 23:09:00.303 [main] INFO c.g.reader.osm.WaySegmentParser - pass1 - processed ways: 20,000,000, accepted ways: 2,945,250, way nodes: 27,219,564, totalMB:102400, usedMB:16244
2024-01-22 23:09:08.947 [main] INFO c.g.reader.osm.WaySegmentParser - pass1 - start reading OSM relations
2024-01-22 23:09:10.450 [main] INFO c.g.reader.osm.WaySegmentParser - pass1 - finished, processed ways: 26,536,431, accepted ways: 4,134,225, way nodes: 34,192,093, relations: 439,042, totalMB:102400, usedMB:38454
2024-01-22 23:09:10.450 [main] INFO c.g.reader.osm.WaySegmentParser - pass1 - finished, took: 1min 37s
2024-01-22 23:09:10.450 [main] INFO c.g.reader.osm.WaySegmentParser - Creating graph. Node count (pillar+tower): 34192093, totalMB:102400, usedMB:38454
2024-01-22 23:09:10.450 [main] INFO c.g.reader.osm.WaySegmentParser - pass2 - start
2024-01-22 23:09:10.470 [main] INFO c.g.reader.osm.WaySegmentParser - pass2 - start reading OSM nodes
2024-01-22 23:09:10.472 [main] INFO c.g.reader.dem.SRTMProvider - srtm Elevation Provider, from: https://srtm.kurviger.de/SRTM3/, to: /opt/graphhopper/production/srtmprovider, as: RAM_STORE using interpolate: false
java.lang.RuntimeException: Could not parse OSM file: /opt/graphhopper/production/canada-latest.osm.pbf
at com.graphhopper.reader.osm.WaySegmentParser.readOSM(WaySegmentParser.java:406)
at com.graphhopper.reader.osm.WaySegmentParser.readOSM(WaySegmentParser.java:107)
at com.graphhopper.reader.osm.OSMReader.readGraph(OSMReader.java:177)
at com.graphhopper.GraphHopper.importOSM(GraphHopper.java:954)
at com.graphhopper.GraphHopper.process(GraphHopper.java:892)
at com.graphhopper.GraphHopper.importAndClose(GraphHopper.java:853)
at com.graphhopper.application.cli.ImportCommand.run(ImportCommand.java:36)
at com.graphhopper.application.cli.ImportCommand.run(ImportCommand.java:27)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:98)
at io.dropwizard.cli.Cli.run(Cli.java:78)
at io.dropwizard.Application.run(Application.java:94)
at com.graphhopper.application.GraphHopperApplication.main(GraphHopperApplication.java:38)
Caused by: java.lang.RuntimeException: There was an issue looking up the coordinates for 52.1287381,-104.530279
at com.graphhopper.reader.dem.AbstractSRTMElevationProvider.updateHeightsFromFile(AbstractSRTMElevationProvider.java:162)
at com.graphhopper.reader.dem.AbstractSRTMElevationProvider.getEle(AbstractSRTMElevationProvider.java:112)
at com.graphhopper.reader.dem.ElevationProvider.getEle(ElevationProvider.java:52)
at com.graphhopper.reader.osm.WaySegmentParser$Pass2Handler.lambda$handleNode$0(WaySegmentParser.java:209)
at com.graphhopper.reader.osm.OSMNodeData.addCoordinatesIfMapped(OSMNodeData.java:158)
at com.graphhopper.reader.osm.WaySegmentParser$Pass2Handler.handleNode(WaySegmentParser.java:209)
at com.graphhopper.reader.osm.WaySegmentParser$ReaderElementHandler.handleElement(WaySegmentParser.java:498)
at com.graphhopper.reader.osm.WaySegmentParser.readOSM(WaySegmentParser.java:401)
... 11 more
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
at java.base/java.util.zip.ZipInputStream.read(ZipInputStream.java:197)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.graphhopper.reader.dem.SRTMProvider.readFile(SRTMProvider.java:128)
at com.graphhopper.reader.dem.AbstractSRTMElevationProvider.updateHeightsFromFile(AbstractSRTMElevationProvider.java:146)
... 18 more
Instead of a manual reproduction it would be even better to have a unit or integration test reproducing the issue.
Expected behavior
A clear and concise description of what you expected to happen.
- A successfull import
System Information
Provide version of Operating system, JVM and GraphHopper
- Linux 22.04
Screenshots & Logs
If applicable, add screenshots and log output.
In case of a routing problem include a link to GraphHopper Maps. Also tell us your expected routing result via additional markers and screenshots.
Tried with latest north-america as well. Same error at the exact same coordinates. As well tried an old version of Canada. and failing as well. https://download.geofabrik.de/north-america/canada-210101.osm.pbf from Jan 1st 2021.
Can you try cleaning up your elevation folder (default /tmp/srtm). It might be that it somehow corrupted one of the files and GraphHopper is unable to unzip it.