new capacity has to be strictly positive with custom encoder for "piste" ways

[Rafał Kuźmiński]

Describe the bug
Europe graph is crashing with custom encoder. I tested it on single country (Poland) and it worked. Every time after 3 billion node I receive crash:

Could not parse OSM file: /var/cache/appdata/europe-latest.osm.pbf at com.graphhopper.reader.osm.WaySegmentParser.readOSM(WaySegmentParser.java:401) at com.graphhopper.reader.osm.WaySegmentParser.readOSM(WaySegmentParser.java:119) at com.graphhopper.reader.osm.OSMReader.readGraph(OSMReader.java:160) at com.graphhopper.GraphHopper.importOSM(GraphHopper.java:695) at com.graphhopper.GraphHopper.process(GraphHopper.java:651) at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:615) at routeFinder.factory.GraphFactory.createGraphHopper(GraphFactory.kt:134) at routeFinder.factory.GraphFactory.access$createGraphHopper(GraphFactory.kt:27) at routeFinder.factory.GraphFactory$createGraph$2.invokeSuspend(GraphFactory.kt:69) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115) at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) Caused by: java.lang.IllegalArgumentException: new capacity has to be strictly positive at com.graphhopper.storage.RAMDataAccess.ensureCapacity(RAMDataAccess.java:66) at com.graphhopper.reader.PillarInfo.ensureNode(PillarInfo.java:58) at com.graphhopper.reader.PillarInfo.setNode(PillarInfo.java:63) at com.graphhopper.reader.osm.OSMNodeData.addPillarNode(OSMNodeData.java:170) at com.graphhopper.reader.osm.OSMNodeData.addCoordinatesIfMapped(OSMNodeData.java:155) at com.graphhopper.reader.osm.WaySegmentParser$Pass2Handler.handleNode(WaySegmentParser.java:221) at com.graphhopper.reader.osm.WaySegmentParser$ReaderElementHandler.handleElement(WaySegmentParser.java:517) at com.graphhopper.reader.osm.WaySegmentParser.readOSM(WaySegmentParser.java:396)

I tried to follow error stack and somehow "nextPillarId" seems to be negative:

 private int addPillarNode(long osmId, double lat, double lon, double ele) {
        pillarNodes.setNode(nextPillarId, lat, lon, ele);
        int id = pillarNodeToId(nextPillarId);
        idsByOsmNodeIds.put(osmId, id);
        nextPillarId++;
        return id;
    }

because then code execution goes to "setNode" and ensureNode:

  @Override
    public void setNode(int nodeId, double lat, double lon, double ele) {
        ensureNode(nodeId);
...

  @Override
    public void ensureNode(int nodeId) {
        long tmp = (long) nodeId * rowSizeInBytes;
        da.ensureCapacity(tmp + rowSizeInBytes);
    }

end code crashes in ensureCapacity:

 @Override
    public boolean ensureCapacity(long bytes) {
        if (bytes < 0)
            throw new IllegalArgumentException("new capacity has to be strictly positive");
 

If bytes are negative then nodeId had to be negative and somehow "nextPillarId" had to be negative as well. "nextPillarId" is integer so maybe there are too many pillars?

To Reproduce

Steps to reproduce the behavior. For examples:

1. use europe.pbf file from geofabric
2. add custom encoder:

class PisteEncoder : AbstractFlagEncoder(4, 1.0, 0) {

    companion object {
        val encoderName = "piste"
    }

    init {
        avgSpeedEnc = DecimalEncodedValueImpl(EncodingManager.getKey(name, "average_speed"), speedBits, speedFactor, false)
        restrictedValues.clear()
        blockFords(false)
    }

    override fun createEncodedValues(registerNewEncodedValue: MutableList<EncodedValue?>) {
        super.createEncodedValues(registerNewEncodedValue)
        registerNewEncodedValue.add(avgSpeedEnc)
    }

    override fun getTransportationMode(): TransportationMode = TransportationMode.FOOT

    override fun getAccess(way: ReaderWay): EncodingManager.Access {
        return EncodingManager.Access.WAY
    }

    override fun applyWayTags(way: ReaderWay?, edge: EdgeIteratorState?) {
    }

    override fun handleWayTags(edgeFlags: IntsRef, way: ReaderWay): IntsRef {
        if (way.isPisteNordic()) {
            avgSpeedEnc.setDecimal(false, edgeFlags, 5.0)
            accessEnc.setBool(false, edgeFlags, way.isPisteNordic())
            accessEnc.setBool(true, edgeFlags, way.isPisteNordic())
        }

        return edgeFlags
    }

    private fun ReaderWay.isPisteNordic(): Boolean {
        return getTag("piste:type") == "nordic"
    }

    override fun getName(): String {
        return encoderName
    }
}

Expected behavior

System Information

I tested it on our currently used version of Graphhopper which is 5.3. We have a lot of custom encoders, so it was hard to quickly update to latest version to test it. JDK 21

[Peter]

We need a small reproducer otherwise we won't be able to investigate it. Also it should target the latest GraphHopper version.

I do not see how the pillar node can be so high for just Europe that the integer overflows .... but you can try to change the version 5.3 and e.g. throw an exception when nextPillarId is negative and/or even better try to merge this PR which avoids early overflow of this integer (integrated in 8.x).

[Rafał Kuźmiński]

I am trying to rewrite project to 8.0 (btw are there any migration guides between major versions? I could not find any) but I'm pretty sure it will work there because 8.0 is using long.

I am also sure that it has to be integer overflow because there is no code that could change "nextPillarId" to negative. Last log with node count supports this:

Creating graph. Node count (pillar+tower): 3021864391, totalMB:204800, usedMB:157422

Given max int as 2147483647 there had to be overflow in pillar or tower and logs shows that pillar is the one

This is Europe file with 7 encoders and elevation data.

[Peter]

There are no migration guides unfortunately. Feel free to document your process in detail and I'll try to suggest improvements etc so that others can use it as a migration guide.

[Peter]

This should be fixed with #2978