parallel-remote-rsync
Script to handle parallel data transfer from a remote server to a local host.
I have created this sript to easely download the results of my simulations from a remote cluster to my local machine. Here, I use GNU Parallel (http://dx.doi.org/10.5281/zenodo.16303) to transfer data using multiple threads and rsync (http://rsync.samba.org/) to find files that need to be transferred, that is, files that have changed in size or in last-modified time.
Here I decided to go for a ssh to list the files to be downloaded, but it is also possible to run rsync using the --dry-run flag to see what files will be affected.
Supposing that the remote directory has the tree structure depicted below. The script will download the files creating automatically the same structure in your local machine.
remote_dir
│ source_code01.f90
│ source_code02.f90
│ ...
│
└────folder1
│ │ result01.out
│ │ result02.out
│ │ ...
│ │ rstfile.001
│ │ rstfile.002
│ │ ...
│ |
│ └────subfolder1
│ │ data01.dat
│ │ data02.dat
│ │ ...
│
└────folder2
│ file021.txt
│ file022.txt
Moreover, in my case, I used to have some files (named rstfile.*) of which only the last N of them needed to be downloaded, where N is a user-defined parameter. This feature is also included in the script.