Giters

xebialabs / overthere

Runs something "Over there"

Home Page:http://www.xebialabs.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Copying file with CIFS is very slow sometimes

zerikv opened this issue · comments

commented

Hello,

I have noticed that in some configurations, copy a SmbFile can be very slow (I mean totaly unusable).

That seems the case when the destination is a VM hosted on the same machine as the source.

If I have understand right, the copy is done by the Guava class ByteStreams in OverthereFileCopier#copyFile.
The copy uses streams provided by SmbFile.
As I understand theses stream are not buffered.

Do you think to wrap the streams by a Buffered streams could increase the copy rate?

Thank you for your advices.

commented

Hi Zerikv,

Hmm, I'm not sure that is the cause of the problem. ByteStreams.copy tries to read a complete buffer of 4K from its inputstream at a time, so that should already take care of the buffering.

Also, why would this only cause local-to-local copies to be slow? What happens when you copy from-local-SMB-to-remote-SMB or from-remote-SMB-to-local-SMB? Or from local-file-to-local-SMB?

Regards, Vincent.

commented

Hi Vincent,

ByteStreams#copy uses a buffer but under the hood it calls InputStream#read(byte[]) which fills this buffer in a loop of read, so this buffer is filled char by char. In the other side, BufferedInputStream uses a native method System.arraycopy to copy bytes between arrays, so it's can be quicker I guess.

The same copy (one file about 80 Mb) has taken only 6 seconds from a source shared folder (on the VM) toward a destination local folder.

As you suggest, I need to do more tests on this issue. I'll keep you informed with the results.

Thank you for your help,

Eric

commented

Finally, I was not able to reproduce this issue with the same VM on another host.

I have done some trivial tests with ByteStreams.copy in order to compare elapsed times
when the FileInputStreams are used directly and the case where they are wrapped with a Buffered I/O Stream.

These tests were to copy one file (about 730Mb)

  1. between 2 local disks
  2. and from a local disk to a network drive.

It just a trivial test, played one time, and results are maybe specific to the network and hardware used,
but in both cases I have noticed at least a gain of 20% when the Buffered wrappers are used ...

commented

Hi Eric,

Today I ran into this exact same issue while copying a 9MB file from a Unix host to a Windows host on the same network. The transfer with Overthere took many minutes, while smbclient took 1 second.

Changing the code in OverthereFileCopier.copyFile() to use buffered input and output streams caused the transfer time in Overthere to drop to that same second!

I'm not sure in what circumstances this bug occurs (e.g. it does not occur on my local Windows images), but thanks for noticing! I am reopening this issue so that I can fix it for the next release of Overthere.

Regards, Vincent.