ReadAheadRemoteFileInputStream

[Björn Karge]

sshj.sftp.RemoteFile.ReadAheadRemoteFileInputStream.read() leads to data overwrites and/or ArrayIndexOutOfBoundsException when len < res.recvLen.

The problem does not occur with regular read() because there, the size of the response is limited by the len argument.

With read ahead, the size of the response is bounded by a previous len argument.

This bug is not trivial to fix, because it relies on the RemoteFile.checkReadResponse swiss army knife, so there is no easy way to buffer partially evaluated response packets.

[Jeroen van Erp]

Hi bkarge,

Do you have a sample or testcase to reproduce (preferably) both behaviours? That would help tremendously in fixing it.

[Björn Karge]

Sir!

You use an ordered request fifo with entries (r+0, l1), (r+1, l2), ...
When read(myBuf,off,lk) is called, you enqueue (r+k, lk) and dequeue r+0, l1
Then you call checkReadResponse, which effectively calls arraycopy(...myBuf, off, l1) because SFTP will return a block of size l1 for the next response (if no problem did occur, but that's still another story)

If l1 > lk, then you overwrite the end of my buffer.

An idea how to fix the problem would be to hide the request fifo behind an InputStream and fill myBuf byte by byte (and skip response packets internally when needed). You don't have to literally use InputStream, it's just to illustrate the technique.

[Björn Karge]

Below is a demonstration of the behaviour

I have submitted a pull request...

import net.schmizz.sshj.SSHClient;
import net.schmizz.sshj.sftp.OpenMode;
import net.schmizz.sshj.sftp.ReadAheadRemoteFileInputStream;
import net.schmizz.sshj.sftp.RemoteFile;
import net.schmizz.sshj.sftp.SFTPEngine;
import net.schmizz.sshj.transport.verification.HostKeyVerifier;

import java.io.InputStream;
import java.security.PublicKey;
import java.util.Arrays;
import java.util.EnumSet;
import java.util.Random;

public class SftpReadAheadTest {
    public static void main(String[] args) {
        try {
            SSHClient ssh = new SSHClient();
            ssh.loadKnownHosts();
            ssh.addHostKeyVerifier(new HostKeyVerifier()
            {
                @Override
                public boolean verify( String hostname, int port, PublicKey key ) {
                    return true;
                }
            });
            ssh.connect("localhost");
            ssh.authPublickey(System.getProperty("user.name"));

            SFTPEngine sftp = new SFTPEngine( ssh ).init();

            RemoteFile rf;
            rf = sftp.open( "/tmp/SftpReadAheadTest.bin", EnumSet.of( OpenMode.WRITE, OpenMode.CREAT ) );
            byte[] data = new byte[8192];
            new Random( 53 ).nextBytes( data );
            data[3072] = 1;
            rf.write( 0, data, 0, data.length );
            rf.close();

            rf = sftp.open( "/tmp/SftpReadAheadTest.bin" );
            InputStream rs = rf.new ReadAheadRemoteFileInputStream(16 /*maxUnconfirmedReads*/);
);
            byte[] test = new byte[4097];
            int n = 0;

            while (n < 2048)
                n += rs.read( test, n, 2048 - n );

            while (n < 3072)
                n += rs.read( test, n, 3072 - n );

            if (test[3072] != 0)
                System.err.println("buffer overrun!");

            n += rs.read( test, n, test.length - n ); // --> ArrayIndexOutOfBoundsException

            byte[] test2 = new byte[data.length];
            System.arraycopy( test, 0, test2, 0, test.length );

            while (n < data.length)
                n += rs.read( test2, n, data.length - n );

            if (Arrays.equals( data, test2 ))
                System.out.println("All is good!");

        }
        catch (Throwable t)
        {
            t.printStackTrace();
        }
    }
}

[Jeroen van Erp]

Fixed by merge of #199