Giters

stephenh / mirror

A tool for real-time, two-way sync for remote (e.g. desktop/laptop) development

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support very large files

stephenh opened this issue · comments

commented

Files over ~500mb / ~1gb give mirror trouble b/c the RPC framework (grpc-java) we use for "really snappy bi-directional streaming" only supports in-memory / on-Java-heap byte[]s for its RPC messages, and doesn't support passing around zero-copy files / memory regions like netty's FileRegion.

So, for now, mirror just can't really send those sort of files unless/maybe you give it a huge heap.

We should probably detect the current size of heap, guess that we can do files ~half that size, and just ignore any file larger than that, which is basically #37.

commented

The best solution will be to change the transfer algorithm for such large files. For example, use rsync for delta-transfers.

Thanks