Documentation for local mounts
martinvelez opened this issue · comments
In my case, I do not need to ssh into remote machines. I am trying to use this tool to copy data between two local NFS mounts. For example, I am trying to copy data from foo to bar.
martin.velez@example.com:~$ time sudo ./parsyncfp --maxload=5.5 --NP=4 --startdir=/mnt/ foo /mnt/bar
Is this tool appropriate for that? Can I disable that ssh check?
On Monday, May 6, 2019 3:54:24 PM PDT Martin Velez wrote:
In my case, I do not need to ssh into remote machines. I am trying to
use this tool to copy data between two local NFS mounts.
Is this tool appropriate for that? Can I disable that ssh check?
You should be able to just allow the ssh check to go forward - it's just adding your own
public ssh key to your ~/.ssh/authorized_keys, so that should make no difference,
functionally.
While I use it routinely to rsync large dir trees back and forth over locally mounted
parallel filesystems, I have seen instances where using it over locally mounted NFS
filesystems seems to cause it to choke.
Please try it and see how it behaves and report back. It may be that the NFS buffering
exposes some inherent logic flaw that I don't understand.
hjm
Harry Mangalam,
Info[1]
…--------
[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html
To clarify:
- foo (source) is a FreeNAS filesystem mounted via NFS.
- bar (destination) is a Lustre filesystem mounted via NFS.
I tried it last night. It only transferred 11 / 4092 folders. I got Argument list too long errors. What causes that? Is there a way to fix it?
martin.velez@example:~$ time sudo ./parsyncfp --startdir=/mnt foo /mnt/bar
INFO: parsyncfp would like to append 'ForwardX11Trusted yes' & 'ForwardX11 yes'
to your ~/.ssh/config.
Skipping this may result in a lot of odd ssh warnings being emitted during
the run if you don't have ssh set correctly for the remote system, but the
transfer should still work.)
If this mod of your ~/.ssh/config file is OK, hit [Enter]. Otherwise hit [s] to skip.
sh: 1: cannot create /home/martin.velez/.ssh/config: Directory nonexistent
chmod: cannot access �/home/martin.velez/.ssh/config�: No such file or directory
INFO: Your ~/.ssh/config file is set correctly.
WARN: About to remove all the old cached chunkfiles from [/home/martin.velez/.parsyncfp/fpcache].
Enter ^C to stop this.
If you specified '--nowait', cache will be cleared in 3s regardless.
Otherwise, hit [Enter] and I'll clear them.
Press [ENTER] to continue.
INFO: The fpart chunk files [/home/martin.velez/.parsyncfp/fpcache/f*] are cleared .. continuing.
INFO: Forking fpart. Check [/home/martin.velez/.parsyncfp/fpcache/fpart.log.02.50.51_2019-05-07] for errors if it hangs.
INFO: Starting the 1st [6] rsyncs ..
| Elapsed | 1m | [ eth0] MB/s | Running || Susp'd | Chunks [2019-05-07]
Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs | [UpTo] of [ToDo]
02.51.08 0.07 0.00 133.87 / 0.00 6 <> 0 [6] of [324]
02.51.11 0.12 0.48 1.36 / 0.00 6 <> 0 [6] of [830]
02.51.14 0.17 1.25 24.91 / 0.00 6 <> 0 [6] of [1696]
02.51.17 0.22 1.25 7.18 / 0.00 6 <> 0 [6] of [2653]
02.51.20 0.27 1.63 68.60 / 0.00 6 <> 0 [9] of [3568]
02.51.23 0.32 1.74 67.58 / 0.00 6 <> 0 [12] of [4437]
02.51.26 0.37 1.74 2.69 / 0.00 6 <> 0 [12] of [5313]
02.51.30 0.43 2.08 28.97 / 0.00 6 <> 0 [13] of [6230]
02.51.33 0.48 2.08 47.77 / 0.00 6 <> 0 [15] of [7226]
02.51.36 0.53 2.47 52.53 / 0.00 6 <> 0 [18] of [8167]
02.51.39 0.58 2.68 15.22 / 0.00 6 <> 0 [18] of [9149]
02.51.43 0.65 2.68 94.44 / 0.00 6 <> 0 [24] of [10212]
02.51.46 0.70 3.18 17.40 / 0.00 6 <> 0 [24] of [11017]
02.51.49 0.75 3.41 43.08 / 0.00 6 <> 0 [26] of [12058]
02.51.53 0.82 3.41 70.81 / 0.00 5 <> 0 [30] of [12927]
02.51.56 0.87 3.54 64.43 / 0.00 6 <> 0 [32] of [14051]
02.51.59 0.92 3.33 62.13 / 0.00 6 <> 0 [36] of [15202]
02.52.03 0.98 3.33 35.77 / 0.00 5 <> 0 [38] of [16367]
02.52.06 1.03 3.39 104.80 / 0.00 5 <> 0 [42] of [17596]
02.52.10 1.10 3.44 96.51 / 0.00 6 <> 0 [44] of [18865]
02.52.13 1.15 3.72 67.11 / 0.00 6 <> 0 [48] of [20224]
| Elapsed | 1m | [ eth0] MB/s | Running || Susp'd | Chunks [2019-05-07]
Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs | [UpTo] of [ToDo]
02.52.17 1.22 3.72 106.09 / 0.00 6 <> 0 [50] of [21520]
02.52.20 1.27 3.66 109.77 / 0.00 4 <> 0 [54] of [22898]
02.52.24 1.33 4.09 35.23 / 0.00 5 <> 0 [56] of [24215]
02.52.27 1.38 4.09 138.17 / 0.00 5 <> 0 [60] of [25602]
02.52.31 1.45 4.16 119.43 / 0.00 6 <> 0 [62] of [26920]
02.52.34 1.50 4.39 22.97 / 0.00 5 <> 0 [63] of [28240]
02.52.38 1.57 4.39 48.21 / 0.00 4 <> 0 [66] of [29645]
02.52.42 1.63 4.44 124.34 / 0.00 6 <> 0 [69] of [31022]
02.52.45 1.68 4.57 126.95 / 0.00 6 <> 0 [72] of [32446]
02.52.49 1.75 4.68 1.35 / 0.00 6 <> 0 [72] of [33720]
02.52.52 1.80 4.68 116.63 / 0.00 6 <> 0 [75] of [35204]
02.52.56 1.87 4.79 91.55 / 0.00 6 <> 0 [78] of [36653]
02.53.00 1.93 4.72 66.25 / 0.00 6 <> 0 [81] of [38087]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too longhead of the transfer..
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.45 4.68 1.24 35.25 / 0.00 3 <> 0 [84] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.48 4.73 1.24 46.57 / 0.00 6 <> 0 [87] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.52 4.80 1.78 48.44 / 0.00 5 <> 0 [90] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.55 4.85 1.80 53.06 / 0.00 6 <> 0 [93] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.55.59 4.92 1.82 47.66 / 0.00 6 <> 0 [96] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.03 4.98 1.82 51.00 / 0.00 5 <> 0 [100] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.06 5.03 2.15 37.64 / 0.00 6 <> 0 [102] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.10 5.10 2.06 63.50 / 0.00 6 <> 0 [106] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
| Elapsed | 1m | [ eth0] MB/s | Running || Susp'd | Chunks [2019-05-07]
Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs | [UpTo] of [ToDo]
sh: 1: ls: Argument list too long
02.56.13 5.15 2.06 35.14 / 0.00 6 <> 0 [108] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.17 5.22 2.37 128.42 / 0.00 6 <> 0 [112] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.20 5.27 2.42 33.67 / 0.00 6 <> 0 [114] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.24 5.33 2.31 114.27 / 0.00 6 <> 0 [118] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.28 5.40 2.31 105.53 / 0.00 6 <> 0 [120] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.31 5.45 2.53 15.30 / 0.00 6 <> 0 [121] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.34 5.50 2.48 42.67 / 0.00 4 <> 0 [124] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.38 5.57 2.48 106.17 / 0.00 5 <> 0 [126] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.41 5.62 2.61 19.61 / 0.00 5 <> 0 [127] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.45 5.68 2.56 49.69 / 0.00 5 <> 0 [130] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.49 5.75 2.59 117.46 / 0.00 6 <> 0 [133] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.52 5.80 2.59 85.25 / 0.00 6 <> 0 [133] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.55 5.85 2.62 47.25 / 0.00 3 <> 0 [136] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.56.59 5.92 2.57 136.76 / 0.00 6 <> 0 [140] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.03 5.98 2.57 91.77 / 0.00 6 <> 0 [142] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.06 6.03 2.69 1.04 / 0.00 6 <> 0 [142] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.09 6.08 2.87 91.61 / 0.00 5 <> 0 [144] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.13 6.15 2.87 95.08 / 0.00 4 <> 0 [146] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.16 6.20 2.96 38.93 / 0.00 6 <> 0 [148] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.20 6.27 3.21 38.89 / 0.00 6 <> 0 [150] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.23 6.32 3.21 91.42 / 0.00 6 <> 0 [152] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
| Elapsed | 1m | [ eth0] MB/s | Running || Susp'd | Chunks [2019-05-07]
Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs | [UpTo] of [ToDo]
sh: 1: ls: Argument list too long
02.57.27 6.38 3.35 0.96 / 0.00 6 <> 0 [152] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.30 6.43 3.56 27.52 / 0.00 6 <> 0 [154] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.34 6.50 3.52 113.40 / 0.00 6 <> 0 [158] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.37 6.55 3.52 53.30 / 0.00 5 <> 0 [159] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.41 6.62 3.64 19.94 / 0.00 6 <> 0 [160] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.44 6.67 3.51 104.16 / 0.00 6 <> 0 [164] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.48 6.73 3.51 73.16 / 0.00 6 <> 0 [166] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.51 6.78 3.87 0.96 / 0.00 6 <> 0 [166] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.55 6.85 3.96 88.17 / 0.00 5 <> 0 [169] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.57.58 6.90 3.96 74.64 / 0.00 5 <> 0 [171] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.02 6.97 4.04 16.97 / 0.00 6 <> 0 [172] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.05 7.02 4.12 69.73 / 0.00 6 <> 0 [176] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.09 7.08 3.95 95.61 / 0.00 6 <> 0 [178] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.12 7.13 3.95 140.59 / 0.00 6 <> 0 [182] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.16 7.20 4.11 50.70 / 0.00 6 <> 0 [184] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.19 7.25 4.18 50.04 / 0.00 6 <> 0 [184] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.23 7.32 4.18 134.90 / 0.00 6 <> 0 [188] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.26 7.38 4.41 32.33 / 0.00 6 <> 0 [190] of [0]
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
sh: 1: ls: Argument list too long
02.58.30 7.43 4.38 45.05 / 0.00 6 <> 0 [190] of [0]
sh: 1: ls: Argument list too long
INFO: Done. Please check the target to make sure
expected files are where they're supposed to be.
INFO:
The parsyncfp cache dir takes up [1.4G /home/martin.velez/.parsyncfp]
Don't forget to delete it, but wait until you are sure that your job
completed correctly, so you don't need the log files anymore.
Reminder: check the parsyncfp log [/home/martin.velez/.parsyncfp/rsync-logfile-02.50.51_2019-05-07_189]
and the fpart log [/home/martin.velez/.parsyncfp/fpcache/fpart.log.02.50.51_2019-05-07]
if there were errors. Use '--verbose=1' for less output.
Thanks for using parsyncfp. Tell me how to make it better.
<hjmangalam@gmail.com>
real 7m42.134s
user 0m19.844s
sys 0m19.428s
On Tuesday, May 7, 2019 9:28:09 AM PDT Martin Velez wrote:
To clarify:
* foo (source) is a FreeNAS filesystem mounted via NFS.
* bar (destination) is a Lustre filesystem mounted via NFS.
I tried it last night. It only transferred 11 / 4092 folders. I got
`Argument list too long` errors. What causes that? Is there a way to
fix it?
Yes. In fact this is a fairly frequent problem that I should address programmatically. It
results when your chunk size is too small, resulting in a huge list of f.* files to transfer (in
your ~/.parsyncfp dir.. Try doubling or tripling the chunk size you're using and let me
know if that improves things.
hjm
…
```
***@***.***:~$ time sudo ./parsyncfp --startdir=/mnt foo
/mnt/bar INFO: parsyncfp would like to append 'ForwardX11Trusted yes' &
'ForwardX11 yes' to your ~/.ssh/config.
Skipping this may result in a lot of odd ssh warnings being
emitted during the run if you don't have ssh set correctly for the
remote system, but the transfer should still work.)
If this mod of your ~/.ssh/config file is OK, hit [Enter]. Otherwise hit
[s] to skip.
sh: 1: cannot create /home/martin.velez/.ssh/config: Directory
nonexistent chmod: cannot access /home/martin.velez/.ssh/config: No
such file or directory INFO: Your ~/.ssh/config file is set correctly.
WARN: About to remove all the old cached chunkfiles from
[/home/martin.velez/.parsyncfp/fpcache]. Enter ^C to stop this.
If you specified '--nowait', cache will be cleared in 3s regardless.
Otherwise, hit [Enter] and I'll clear them.
Press [ENTER] to continue.
INFO: The fpart chunk files [/home/martin.velez/.parsyncfp/fpcache/f*]
are cleared .. continuing. INFO: Forking fpart. Check
[/home/martin.velez/.parsyncfp/fpcache/fpart.log.02.50.51_2019-05-07]
for errors if it hangs. INFO: Starting the 1st [6] rsyncs ..
| Elapsed | 1m | [ eth0] MB/s | Running || Susp'd |
| Chunks [2019-05-07]
Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs |
[UpTo] of [ToDo] 02.51.08 0.07 0.00 133.87 / 0.00
6 <> 0 [6] of [324] 02.51.11 0.12 0.48 1.36 /
0.00 6 <> 0 [6] of [830] 02.51.14 0.17
1.25 24.91 / 0.00 6 <> 0 [6] of [1696]
02.51.17 0.22 1.25 7.18 / 0.00 6 <> 0
[6] of [2653] 02.51.20 0.27 1.63 68.60 / 0.00
6 <> 0 [9] of [3568] 02.51.23 0.32 1.74 67.58
/ 0.00 6 <> 0 [12] of [4437] 02.51.26 0.37
1.74 2.69 / 0.00 6 <> 0 [12] of
[5313] 02.51.30 0.43 2.08 28.97 / 0.00 6 <>
0 [13] of [6230] 02.51.33 0.48 2.08 47.77 / 0.00
6 <> 0 [15] of [7226] 02.51.36 0.53 2.47
52.53 / 0.00 6 <> 0 [18] of [8167] 02.51.39
0.58 2.68 15.22 / 0.00 6 <> 0 [18]
of [9149] 02.51.43 0.65 2.68 94.44 / 0.00 6
<> 0 [24] of [10212] 02.51.46 0.70 3.18 17.40 /
0.00 6 <> 0 [24] of [11017] 02.51.49 0.75
3.41 43.08 / 0.00 6 <> 0 [26] of
[12058] 02.51.53 0.82 3.41 70.81 / 0.00 5 <>
0 [30] of [12927] 02.51.56 0.87 3.54 64.43 / 0.00
6 <> 0 [32] of [14051] 02.51.59 0.92
3.33 62.13 / 0.00 6 <> 0 [36] of [15202]
02.52.03 0.98 3.33 35.77 / 0.00 5 <> 0
[38] of [16367] 02.52.06 1.03 3.39 104.80 / 0.00
5 <> 0 [42] of [17596] 02.52.10 1.10 3.44
96.51 / 0.00 6 <> 0 [44] of [18865] 02.52.13
1.15 3.72 67.11 / 0.00 6 <> 0 [48]
of [20224]
| Elapsed | 1m | [ eth0] MB/s | Running || Susp'd |
| Chunks [2019-05-07]
Time | time(m) | Load | TCP / RDMA out | PIDs || PIDs |
[UpTo] of [ToDo] 02.52.17 1.22 3.72 106.09 / 0.00
6 <> 0 [50] of [21520] 02.52.20 1.27 3.66
109.77 / 0.00 4 <> 0 [54] of [22898] 02.52.24
1.33 4.09 35.23 / 0.00 5 <> 0 [56]
of [24215] 02.52.27 1.38 4.09 138.17 / 0.00 5
<> 0 [60] of [25602] 02.52.31 1.45 4.16 119.43 /
0.00 6 <> 0 [62] of [26920] 02.52.34 1.50
4.39 22.97 / 0.00 5 <> 0 [63] of
[28240] 02.52.38 1.57 4.39 48.21 / 0.00 4 <>
0 [66] of [29645] 02.52.42 1.63 4.44 124.34 / 0.00
6 <> 0 [69] of [31022] 02.52.45 1.68
You probably need to specify --chunkksize to a larger value.
I tried 3 values: --chunkksize 10G, --chunkksize 100G and --chunkksize 1T. I still see those errors.
On Wednesday, June 5, 2019 3:52:50 PM PDT Martin Velez wrote:
I tried 3 values: `--chunkksize 10G`, `--chunkksize 100G` and
`--chunkksize 1T`. I still see those errors.
Could you send me the command you used and (assuming that you're using
~/.parsyncfp) attach the file:
*~/.parsyncfp/fpcache/*fpart.log ... (whatever is the one that failed most recently.
Obviously, look at it to make sure it doesn't contain any proprietary info, but it should
contain only the stats from the fpart chunking operation.
gzip it if it's ginormous.
hjm
Harry Mangalam,
Info[1]
…--------
[1] http://moo.nac.uci.edu/~hjm/hjm.sig.html
Since I opened this ticket, I tried https://github.com/jbd/msrsync and it helped me speed up copying large directories.
Hi Martin,
Glad that worked for you. I haven't tried it, but I will. Can you say
what you liked about it vs parsyncfp or fpsync or any of the others?
Was it easier to use? faster? less overhead? better docs?
Harry
…On Tue, Jul 7, 2020 at 7:04 PM Martin Velez ***@***.***> wrote:
Since I opened this ticket, I tried https://github.com/jbd/msrsync and it
helped me speed up my file transfer.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#21 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AASF3YYSIYDXJ3W2JBNBCFTR2PHY7ANCNFSM4HLEAAKA>
.
--
Harry Mangalam
It was easier to use. I was able to use it with directories of any size without having to find the right parameters by trial-and-error. The only parameter I tuned was the number of parallel processes. I was never able to compare speed.