Sorting error prevents I2P-Bote from deleting email packets (Trac #1931)

[Jack Grigg]

[str4d: This was sent to me December 2015 via I2P-Bote, but I didn't get a ticket created for it at the time.]

There was a problem with my old address, I was able to send mails but I havent received anything. I was trying to send mails to self without success.

I2P version: 0.9.23-0
Java version: Oracle Corporation 1.8.0_66 (Java(TM) SE Runtime Environment 1.8.0_66-b18)
Wrapper version: 3.5.25
Server version: 8.1.17.v20150415
Servlet version: Jasper JSP 2.1 Engine
Platform: Windows 7 x86 6.1

15.4.12 14:32:43 ERROR [ExpiratnThrd] .bote.service.ExpirationThread: Exception caught in ExpirationThread loop
java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeHi(Unknown Source)
at java.util.TimSort.mergeAt(Unknown Source)
at java.util.TimSort.mergeForceCollapse(Unknown Source)
at java.util.TimSort.sort(Unknown Source)
at java.util.Arrays.sort(Unknown Source)
at i2p.bote.folder.Folder.getFilenames(Folder.java:98)
at i2p.bote.folder.DeletionAwareDhtFolder.getFilenames(DeletionAwareDhtFolder.java:127)
at i2p.bote.folder.Folder.iterate(Folder.java:125)
at i2p.bote.folder.PacketFolder$1.<init>(PacketFolder.java:97)
at i2p.bote.folder.PacketFolder.iterator(PacketFolder.java:96)
at i2p.bote.folder.EmailPacketFolder.deleteExpired(EmailPacketFolder.java:142)
at i2p.bote.service.ExpirationThread.run(ExpirationThread.java:53)

Migrated from https://trac.i2p2.de/ticket/1931

{
    "status": "closed", 
    "changetime": "2017-01-15T22:32:09", 
    "description": "[str4d: This was sent to me December 2015 via I2P-Bote, but I didn't get a ticket created for it at the time.]\n\nThere was a problem with my old address, I was able to send mails but I havent received anything. I was trying to send mails to self without success.\n\nI2P version: 0.9.23-0\nJava version: Oracle Corporation 1.8.0_66 (Java(TM) SE Runtime Environment 1.8.0_66-b18)\nWrapper version: 3.5.25\nServer version: 8.1.17.v20150415\nServlet version: Jasper JSP 2.1 Engine\nPlatform: Windows 7 x86 6.1\n\n{{{\n15.4.12 14:32:43 ERROR [ExpiratnThrd] .bote.service.ExpirationThread: Exception caught in ExpirationThread loop\njava.lang.IllegalArgumentException: Comparison method violates its general contract!\nat java.util.TimSort.mergeHi(Unknown Source)\nat java.util.TimSort.mergeAt(Unknown Source)\nat java.util.TimSort.mergeForceCollapse(Unknown Source)\nat java.util.TimSort.sort(Unknown Source)\nat java.util.Arrays.sort(Unknown Source)\nat i2p.bote.folder.Folder.getFilenames(Folder.java:98)\nat i2p.bote.folder.DeletionAwareDhtFolder.getFilenames(DeletionAwareDhtFolder.java:127)\nat i2p.bote.folder.Folder.iterate(Folder.java:125)\nat i2p.bote.folder.PacketFolder$1.<init>(PacketFolder.java:97)\nat i2p.bote.folder.PacketFolder.iterator(PacketFolder.java:96)\nat i2p.bote.folder.EmailPacketFolder.deleteExpired(EmailPacketFolder.java:142)\nat i2p.bote.service.ExpirationThread.run(ExpirationThread.java:53)\n}}}", 
    "reporter": "lv-server", 
    "cc": "", 
    "resolution": "fixed", 
    "_ts": "1484519529131204", 
    "component": "apps/plugins", 
    "summary": "Sorting error prevents I2P-Bote from deleting email packets", 
    "priority": "minor", 
    "keywords": "I2P-Bote", 
    "version": "0.9.23", 
    "parents": "", 
    "time": "2017-01-15T18:24:10", 
    "milestone": "0.9.29", 
    "owner": "str4d", 
    "type": "defect"
}

[Jack Grigg]

Trac update at 20170115T18:59:00: str4d commented:

I'm guessing this was in I2P-Bote 0.4.1 or 0.4.2.

Similar to https://trac.i2p2.de/ticket/811. The relevant code here is:

    protected File[] getFilenames() {
        File[] files = storageDir.listFiles(new FilenameFilter() {
            @Override
            public boolean accept(File dir, String name) {
                return name.toUpperCase().endsWith(fileExtension.toUpperCase());
            }
        });
        if (files == null) {
            log.error("Cannot list files in directory <" + storageDir + ">");
            files = new File[0];
        } else
            // sort files by date, newest first
            Arrays.sort(files, new Comparator<File>() {
                @Override
                public int compare(File f1, File f2) {
                    return (int)Math.signum(f2.lastModified() - f1.lastModified());
                }
            });

        return files;
    }

File.lastModified() should always return either the time of last modification, or 0. So what is likely the problem is that the packets in the folder were being updated at the same time as this was running, making File.lastModified() non-deterministic.

The ordering of packets is non-critical (ordering newest-first is AFAICT just an optimisation for generally needing the latest packets), so I will solve this in the same fashion as https://trac.i2p2.de/ticket/811.

[Jack Grigg]

Trac update at 20170115T19:11:20: str4d commented:

Fixed in d6191ad2c056f0014f468023cfe115b1cd21baf4.

[Jack Grigg]

Trac update at 20170115T21:15:40:

• zzz commented:

This is a Java 8 thing, where the sort algo changed. Happened years ago over in router code, where it was exceedingly rare, and I just fixed it by catching IAE and retrying (or not, can't recall).

• zzz changed owner from "" to "str4d"
• zzz changed status from "new" to "assigned"

[Jack Grigg]

Trac update at 20170115T22:31:59:

• str4d commented:

I'm not bothered with retrying, because the code calling this wants to iterate over the whole folder, so ordering doesn't matter.

• str4d changed resolution from "" to "fixed"
• str4d changed status from "assigned" to "closed"

[Jack Grigg]

Trac update at 20170115T22:32:09: str4d changed milestone from "undecided" to "0.9.29"