Feature request: Round Robin on Available Interfaces
rapier1 opened this issue · comments
It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.
Thanks!
thanks for the suggestion,
Do you mean can it use bonded interfaces? Or if there are multiple
interfaces with multiple routes, it can use both?
Right now, pfp detects multiple interfaces (for the monitoring) and insists
that you specify which one you want to watch.
It currently sends data thru the interface that your routing provides as
the first path to the target.
I'm currently working on the multihost version, so it can send data thru
multiple hosts to (optionally) multiple receive hosts, each with different
receive paths). That version should be ready for testing in about a week or
so. Is that closer to what you want?
harry
…On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***> wrote:
It seems that parsyncfp won't make use of multiple interfaces on a
multihomed host. Before I start looking at doing myself I was wondering if
that was on your radar or if that functionality already exists and I'm just
missing it.
Thanks!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#40>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ>
.
--
Harry Mangalam
And if it's really a round-robin use, how would you want the bytes to
actually be allocated? I'm trying to think of a way that an actual RR send
would work, especially with the routing.. Can you elaborate? Or give me an
example?
h
…On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***> wrote:
It seems that parsyncfp won't make use of multiple interfaces on a
multihomed host. Before I start looking at doing myself I was wondering if
that was on your radar or if that functionality already exists and I'm just
missing it.
Thanks!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#40>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ>
.
--
Harry Mangalam
Really, I'm just thinking about there being a list of interfaces and
each rsync picking an interface from that list when it's instantiated.
Nothing on a byte level. Basically, we have a situation in our
datacenter where we have a host with 8x100Gb interfaces. All/most of the
interfaces are in the same CIDR block. So we were thinking that having
parsyncfp cycle through the interfaces would help with throughput
(assuming that the storage can keep up).
Also, I want to say how helpful parsyncfp has been. We just moved around
3 or 4 PB of data from our old storage system to a new one. Lustre (of
course) was the big drag but parsyncfp worked great. I actually built a
front end to it so that users could easily build the parsyncfp command
and submit them as slurm jobs. I had to modify parsyncfp a bit - some of
those changes might be useful so I'll submit a pull request once I get
my version merged with HEAD.
…On 3/24/21 11:36 AM, Harry Mangalam wrote:
And if it's really a round-robin use, how would you want the bytes to
actually be allocated? I'm trying to think of a way that an actual RR send
would work, especially with the routing.. Can you elaborate? Or give me an
example?
h
On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
wrote:
> It seems that parsyncfp won't make use of multiple interfaces on a
> multihomed host. Before I start looking at doing myself I was
wondering if
> that was on your radar or if that functionality already exists and
I'm just
> missing it.
>
> Thanks!
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#40>, or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ>
> .
>
--
Harry Mangalam
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ>.
so you want pfp to handle the detection and use of the rsync flag:
--address=ADDRESS bind address for outgoing socket to daemon
This would be on the same host, right?
So when each new chunk is started up, it chooses its interface from the
list of available?/given? ones? I'd guess you'd probably want to specify
the list so as not to bugger those that are being used for other purposes.
So something like:
…--roundrobin="192.168.0.4,
192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"
If not that, let me know what you're thinking.. The above doesn't sound
too hard. Just a slight mod of the flags to each rsync... altho I could
easily be missing something... Tho monitoring all those interfaces would be
a nightmare...
hjm
On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier ***@***.***>
wrote:
Really, I'm just thinking about there being a list of interfaces and
each rsync picking an interface from that list when it's instantiated.
Nothing on a byte level. Basically, we have a situation in our
datacenter where we have a host with 8x100Gb interfaces. All/most of the
interfaces are in the same CIDR block. So we were thinking that having
parsyncfp cycle through the interfaces would help with throughput
(assuming that the storage can keep up).
Also, I want to say how helpful parsyncfp has been. We just moved around
3 or 4 PB of data from our old storage system to a new one. Lustre (of
course) was the big drag but parsyncfp worked great. I actually built a
front end to it so that users could easily build the parsyncfp command
and submit them as slurm jobs. I had to modify parsyncfp a bit - some of
those changes might be useful so I'll submit a pull request once I get
my version merged with HEAD.
On 3/24/21 11:36 AM, Harry Mangalam wrote:
> And if it's really a round-robin use, how would you want the bytes to
> actually be allocated? I'm trying to think of a way that an actual RR
send
> would work, especially with the routing.. Can you elaborate? Or give me
an
> example?
> h
>
> On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
> wrote:
>
> > It seems that parsyncfp won't make use of multiple interfaces on a
> > multihomed host. Before I start looking at doing myself I was
> wondering if
> > that was on your radar or if that functionality already exists and
> I'm just
> > missing it.
> >
> > Thanks!
> >
> > —
> > You are receiving this because you are subscribed to this thread.
> > Reply to this email directly, view it on GitHub
> > <#40>, or unsubscribe
> >
> <
https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ
>
> > .
> >
>
>
> --
>
> Harry Mangalam
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <
#40 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ
>.
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ>
.
--
Harry Mangalam
I believe that is what we are looking for. Monitoring could be a pain
but one of my colleagues is looking at that. I'll ask him what his
thoughts are.
Chris
…On 3/24/21 2:24 PM, Harry Mangalam wrote:
so you want pfp to handle the detection and use of the rsync flag:
--address=ADDRESS bind address for outgoing socket to daemon
This would be on the same host, right?
So when each new chunk is started up, it chooses its interface from the
list of available?/given? ones? I'd guess you'd probably want to specify
the list so as not to bugger those that are being used for other purposes.
So something like:
--roundrobin="192.168.0.4,
192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"
If not that, let me know what you're thinking.. The above doesn't sound
too hard. Just a slight mod of the flags to each rsync... altho I could
easily be missing something... Tho monitoring all those interfaces would be
a nightmare...
hjm
On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier ***@***.***>
wrote:
> Really, I'm just thinking about there being a list of interfaces and
> each rsync picking an interface from that list when it's instantiated.
> Nothing on a byte level. Basically, we have a situation in our
> datacenter where we have a host with 8x100Gb interfaces. All/most of the
> interfaces are in the same CIDR block. So we were thinking that having
> parsyncfp cycle through the interfaces would help with throughput
> (assuming that the storage can keep up).
>
> Also, I want to say how helpful parsyncfp has been. We just moved around
> 3 or 4 PB of data from our old storage system to a new one. Lustre (of
> course) was the big drag but parsyncfp worked great. I actually built a
> front end to it so that users could easily build the parsyncfp command
> and submit them as slurm jobs. I had to modify parsyncfp a bit - some of
> those changes might be useful so I'll submit a pull request once I get
> my version merged with HEAD.
>
> On 3/24/21 11:36 AM, Harry Mangalam wrote:
> > And if it's really a round-robin use, how would you want the bytes to
> > actually be allocated? I'm trying to think of a way that an actual RR
> send
> > would work, especially with the routing.. Can you elaborate? Or give me
> an
> > example?
> > h
> >
> > On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
> > wrote:
> >
> > > It seems that parsyncfp won't make use of multiple interfaces on a
> > > multihomed host. Before I start looking at doing myself I was
> > wondering if
> > > that was on your radar or if that functionality already exists and
> > I'm just
> > > missing it.
> > >
> > > Thanks!
> > >
> > > —
> > > You are receiving this because you are subscribed to this thread.
> > > Reply to this email directly, view it on GitHub
> > > <#40>, or unsubscribe
> > >
> > <
>
https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ
> >
> > > .
> > >
> >
> >
> > --
> >
> > Harry Mangalam
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub
> > <
>
#40 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ
> >.
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
>
<#40 (comment)>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ>
> .
>
--
Harry Mangalam
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ>.
So I talked to my coworker and he basically hacked getavgnetbw and used
a wildcard to use ib* instead of ${NETIF}. That's dirty hack though. We
talked about it a bit and it's hard because the interface nomenclature
doesn't necessarily work with that outside of infiniband interfaces.
Unless you specifically ask the user to enter a regex or wildcard
solution that would work.
However, if that can be solved it should be possible to just loop
through the results of the cat and aggregate the values of
/sys/class/net/[ifname]*/statistics/rx_bytes. Or even
/sys/class/net/[ifname]*/statistics/rx_bytes | paste -sd+ - | bc would
give you the total across all interfaces in a single command.
He also said that if you make the --interface able to take a list of
interfaces you could just loop over that list. Which would be require
collecting the values in an array or hash. Which isn't awful as long as
you report aggregate values and not per interface. If that sounds like
it might work let me know and I'll see about getting you a patch. Might
take some time as I'm floored with other work right now.
Chris
…On 3/24/21 2:24 PM, Harry Mangalam wrote:
so you want pfp to handle the detection and use of the rsync flag:
--address=ADDRESS bind address for outgoing socket to daemon
This would be on the same host, right?
So when each new chunk is started up, it chooses its interface from the
list of available?/given? ones? I'd guess you'd probably want to specify
the list so as not to bugger those that are being used for other purposes.
So something like:
--roundrobin="192.168.0.4,
192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"
If not that, let me know what you're thinking.. The above doesn't sound
too hard. Just a slight mod of the flags to each rsync... altho I could
easily be missing something... Tho monitoring all those interfaces would be
a nightmare...
hjm
On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier ***@***.***>
wrote:
> Really, I'm just thinking about there being a list of interfaces and
> each rsync picking an interface from that list when it's instantiated.
> Nothing on a byte level. Basically, we have a situation in our
> datacenter where we have a host with 8x100Gb interfaces. All/most of the
> interfaces are in the same CIDR block. So we were thinking that having
> parsyncfp cycle through the interfaces would help with throughput
> (assuming that the storage can keep up).
>
> Also, I want to say how helpful parsyncfp has been. We just moved around
> 3 or 4 PB of data from our old storage system to a new one. Lustre (of
> course) was the big drag but parsyncfp worked great. I actually built a
> front end to it so that users could easily build the parsyncfp command
> and submit them as slurm jobs. I had to modify parsyncfp a bit - some of
> those changes might be useful so I'll submit a pull request once I get
> my version merged with HEAD.
>
> On 3/24/21 11:36 AM, Harry Mangalam wrote:
> > And if it's really a round-robin use, how would you want the bytes to
> > actually be allocated? I'm trying to think of a way that an actual RR
> send
> > would work, especially with the routing.. Can you elaborate? Or give me
> an
> > example?
> > h
> >
> > On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
> > wrote:
> >
> > > It seems that parsyncfp won't make use of multiple interfaces on a
> > > multihomed host. Before I start looking at doing myself I was
> > wondering if
> > > that was on your radar or if that functionality already exists and
> > I'm just
> > > missing it.
> > >
> > > Thanks!
> > >
> > > —
> > > You are receiving this because you are subscribed to this thread.
> > > Reply to this email directly, view it on GitHub
> > > <#40>, or unsubscribe
> > >
> > <
>
https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ
> >
> > > .
> > >
> >
> >
> > --
> >
> > Harry Mangalam
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub
> > <
>
#40 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ
> >.
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
>
<#40 (comment)>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ>
> .
>
--
Harry Mangalam
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ>.
parsyncfp might be thought of as nothing more than a massive hack of rsync,
so more hacks would not be out of line.
I've more or less hacked in the changes required to add multiple interfaces
in the (multihost) version that I'm working on now (tho the 2 are
incompatible) and they should be ready to test within a week. I'll send
them out to you when they are, prior to sending it to github.
Monitoring the bandwidth isnt so much the problem - pfp already does that
with perfquery (on IB) and other tickles, but organizing the returned
output might be tricky. I'll play around with it and see what blows up in
my face ;)
hjm
On Wed, Mar 24, 2021 at 12:52 PM Chris Rapier ***@***.***>
wrote:
… So I talked to my coworker and he basically hacked getavgnetbw and used
a wildcard to use ib* instead of ${NETIF}. That's dirty hack though. We
talked about it a bit and it's hard because the interface nomenclature
doesn't necessarily work with that outside of infiniband interfaces.
Unless you specifically ask the user to enter a regex or wildcard
solution that would work.
However, if that can be solved it should be possible to just loop
through the results of the cat and aggregate the values of
/sys/class/net/[ifname]*/statistics/rx_bytes. Or even
/sys/class/net/[ifname]*/statistics/rx_bytes | paste -sd+ - | bc would
give you the total across all interfaces in a single command.
He also said that if you make the --interface able to take a list of
interfaces you could just loop over that list. Which would be require
collecting the values in an array or hash. Which isn't awful as long as
you report aggregate values and not per interface. If that sounds like
it might work let me know and I'll see about getting you a patch. Might
take some time as I'm floored with other work right now.
Chris
On 3/24/21 2:24 PM, Harry Mangalam wrote:
> so you want pfp to handle the detection and use of the rsync flag:
> --address=ADDRESS bind address for outgoing socket to daemon
>
> This would be on the same host, right?
>
> So when each new chunk is started up, it chooses its interface from the
> list of available?/given? ones? I'd guess you'd probably want to specify
> the list so as not to bugger those that are being used for other
purposes.
> So something like:
>
> --roundrobin="192.168.0.4,
>
192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"
>
> If not that, let me know what you're thinking.. The above doesn't sound
> too hard. Just a slight mod of the flags to each rsync... altho I could
> easily be missing something... Tho monitoring all those interfaces would
be
> a nightmare...
>
> hjm
>
>
>
> On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier ***@***.***>
> wrote:
>
> > Really, I'm just thinking about there being a list of interfaces and
> > each rsync picking an interface from that list when it's instantiated.
> > Nothing on a byte level. Basically, we have a situation in our
> > datacenter where we have a host with 8x100Gb interfaces. All/most of
the
> > interfaces are in the same CIDR block. So we were thinking that having
> > parsyncfp cycle through the interfaces would help with throughput
> > (assuming that the storage can keep up).
> >
> > Also, I want to say how helpful parsyncfp has been. We just moved
around
> > 3 or 4 PB of data from our old storage system to a new one. Lustre (of
> > course) was the big drag but parsyncfp worked great. I actually built a
> > front end to it so that users could easily build the parsyncfp command
> > and submit them as slurm jobs. I had to modify parsyncfp a bit - some
of
> > those changes might be useful so I'll submit a pull request once I get
> > my version merged with HEAD.
> >
> > On 3/24/21 11:36 AM, Harry Mangalam wrote:
> > > And if it's really a round-robin use, how would you want the bytes to
> > > actually be allocated? I'm trying to think of a way that an actual RR
> > send
> > > would work, especially with the routing.. Can you elaborate? Or give
me
> > an
> > > example?
> > > h
> > >
> > > On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
> > > wrote:
> > >
> > > > It seems that parsyncfp won't make use of multiple interfaces on a
> > > > multihomed host. Before I start looking at doing myself I was
> > > wondering if
> > > > that was on your radar or if that functionality already exists and
> > > I'm just
> > > > missing it.
> > > >
> > > > Thanks!
> > > >
> > > > —
> > > > You are receiving this because you are subscribed to this thread.
> > > > Reply to this email directly, view it on GitHub
> > > > <#40>, or
unsubscribe
> > > >
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ
> > >
> > > > .
> > > >
> > >
> > >
> > > --
> > >
> > > Harry Mangalam
> > >
> > > —
> > > You are receiving this because you authored the thread.
> > > Reply to this email directly, view it on GitHub
> > > <
> >
> #40 (comment)
>,
> >
> > > or unsubscribe
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ
> > >.
> > >
> >
> > —
> > You are receiving this because you commented.
> > Reply to this email directly, view it on GitHub
> >
> <
#40 (comment)>,
> > or unsubscribe
> >
> <
https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ
>
> > .
> >
>
>
> --
>
> Harry Mangalam
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <
#40 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ
>.
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AASF3YZFWNDZ6KZWUM6U2Y3TFI7G3ANCNFSM4ZXPJEHQ>
.
--
Harry Mangalam
Cool. I came up with something - 1st pass and not tested but for
getavgnetbw something like this should work.
my $if_hash;
my @if_list = split(/,/, $NETIF);
foreach my $if ***@***.***_list) {
$if = trim($if);
$if_hash{if}[0] = `cat /sys/class/net/$if/statistics/rx_bytes`;
$if_hash{if}[1] = `cat /sys/class/net/$if/statistics/tx_bytes`;
}
sleep $CHECKPERIOD;
foreach my $if ***@***.***_list) {
$if = trim($if);
$rx_sum += `cat /sys/class/net/$if/statistics/rx_bytes` -
$if_hash{$if}[0];
$tx_sum += `cat /sys/class/net/$if/statistics/tx_bytes` -
$if_hash{$if}[1];
}
$avgrec = $rx_sum / $CHECKPERIOD;
$avgtrans = $tx_sum / $CHECKPERIOD;
…On 3/24/21 4:13 PM, Harry Mangalam wrote:
parsyncfp might be thought of as nothing more than a massive hack of rsync,
so more hacks would not be out of line.
I've more or less hacked in the changes required to add multiple interfaces
in the (multihost) version that I'm working on now (tho the 2 are
incompatible) and they should be ready to test within a week. I'll send
them out to you when they are, prior to sending it to github.
Monitoring the bandwidth isnt so much the problem - pfp already does that
with perfquery (on IB) and other tickles, but organizing the returned
output might be tricky. I'll play around with it and see what blows up in
my face ;)
hjm
On Wed, Mar 24, 2021 at 12:52 PM Chris Rapier ***@***.***>
wrote:
> So I talked to my coworker and he basically hacked getavgnetbw and used
> a wildcard to use ib* instead of ${NETIF}. That's dirty hack though. We
> talked about it a bit and it's hard because the interface nomenclature
> doesn't necessarily work with that outside of infiniband interfaces.
> Unless you specifically ask the user to enter a regex or wildcard
> solution that would work.
>
> However, if that can be solved it should be possible to just loop
> through the results of the cat and aggregate the values of
> /sys/class/net/[ifname]*/statistics/rx_bytes. Or even
> /sys/class/net/[ifname]*/statistics/rx_bytes | paste -sd+ - | bc would
> give you the total across all interfaces in a single command.
>
> He also said that if you make the --interface able to take a list of
> interfaces you could just loop over that list. Which would be require
> collecting the values in an array or hash. Which isn't awful as long as
> you report aggregate values and not per interface. If that sounds like
> it might work let me know and I'll see about getting you a patch. Might
> take some time as I'm floored with other work right now.
>
> Chris
>
>
> On 3/24/21 2:24 PM, Harry Mangalam wrote:
> > so you want pfp to handle the detection and use of the rsync flag:
> > --address=ADDRESS bind address for outgoing socket to daemon
> >
> > This would be on the same host, right?
> >
> > So when each new chunk is started up, it chooses its interface from the
> > list of available?/given? ones? I'd guess you'd probably want to
specify
> > the list so as not to bugger those that are being used for other
> purposes.
> > So something like:
> >
> > --roundrobin="192.168.0.4,
> >
>
192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"
> >
> > If not that, let me know what you're thinking.. The above doesn't sound
> > too hard. Just a slight mod of the flags to each rsync... altho I could
> > easily be missing something... Tho monitoring all those interfaces
would
> be
> > a nightmare...
> >
> > hjm
> >
> >
> >
> > On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier ***@***.***>
> > wrote:
> >
> > > Really, I'm just thinking about there being a list of interfaces and
> > > each rsync picking an interface from that list when it's
instantiated.
> > > Nothing on a byte level. Basically, we have a situation in our
> > > datacenter where we have a host with 8x100Gb interfaces. All/most of
> the
> > > interfaces are in the same CIDR block. So we were thinking that
having
> > > parsyncfp cycle through the interfaces would help with throughput
> > > (assuming that the storage can keep up).
> > >
> > > Also, I want to say how helpful parsyncfp has been. We just moved
> around
> > > 3 or 4 PB of data from our old storage system to a new one.
Lustre (of
> > > course) was the big drag but parsyncfp worked great. I actually
built a
> > > front end to it so that users could easily build the parsyncfp
command
> > > and submit them as slurm jobs. I had to modify parsyncfp a bit - some
> of
> > > those changes might be useful so I'll submit a pull request once
I get
> > > my version merged with HEAD.
> > >
> > > On 3/24/21 11:36 AM, Harry Mangalam wrote:
> > > > And if it's really a round-robin use, how would you want the
bytes to
> > > > actually be allocated? I'm trying to think of a way that an
actual RR
> > > send
> > > > would work, especially with the routing.. Can you elaborate? Or
give
> me
> > > an
> > > > example?
> > > > h
> > > >
> > > > On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
> > > > wrote:
> > > >
> > > > > It seems that parsyncfp won't make use of multiple interfaces
on a
> > > > > multihomed host. Before I start looking at doing myself I was
> > > > wondering if
> > > > > that was on your radar or if that functionality already
exists and
> > > > I'm just
> > > > > missing it.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > —
> > > > > You are receiving this because you are subscribed to this thread.
> > > > > Reply to this email directly, view it on GitHub
> > > > > <#40>, or
> unsubscribe
> > > > >
> > > > <
> > >
> >
>
https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ
> > > >
> > > > > .
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Harry Mangalam
> > > >
> > > > —
> > > > You are receiving this because you authored the thread.
> > > > Reply to this email directly, view it on GitHub
> > > > <
> > >
> >
#40 (comment)
> >,
> > >
> > > > or unsubscribe
> > > > <
> > >
> >
>
https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ
> > > >.
> > > >
> > >
> > > —
> > > You are receiving this because you commented.
> > > Reply to this email directly, view it on GitHub
> > >
> > <
>
#40 (comment)>,
> > > or unsubscribe
> > >
> > <
>
https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ
> >
> > > .
> > >
> >
> >
> > --
> >
> > Harry Mangalam
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub
> > <
>
#40 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ
> >.
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
>
<#40 (comment)>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AASF3YZFWNDZ6KZWUM6U2Y3TFI7G3ANCNFSM4ZXPJEHQ>
> .
>
--
Harry Mangalam
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAKL66CBFBCVSI6KWMWLKADTFJBUZANCNFSM4ZXPJEHQ>.
Hi Chris,
that's a solution close to my heart. :) - it does look quite a bit like
what's in there already. It's the cycling thru the different rysncs and
(mostly) how to present it in a coherent manner in the interface, but since
each rsync will be talking thru it's own interface and can be told to
monitor THAT interface, it may already be solved by just modifying the args
to rsync. (ie feeding it the
--address=xxx.xxx.xxx.xxx option and translating the interface name to the
--interface-=ib0/eth0/etc. I would then have to capture and combine all
the rsync outputs to provide the summary line at each checkperiod. (ugly
but eminently doable..).
Let me get the actual roundrobin routine working first.
hjm
On Wed, Mar 24, 2021 at 1:18 PM Chris Rapier ***@***.***>
wrote:
… Cool. I came up with something - 1st pass and not tested but for
getavgnetbw something like this should work.
my $if_hash;
my @if_list = split(/,/, $NETIF);
foreach my $if ***@***.***_list) {
$if = trim($if);
$if_hash{if}[0] = `cat /sys/class/net/$if/statistics/rx_bytes`;
$if_hash{if}[1] = `cat /sys/class/net/$if/statistics/tx_bytes`;
}
sleep $CHECKPERIOD;
foreach my $if ***@***.***_list) {
$if = trim($if);
$rx_sum += `cat /sys/class/net/$if/statistics/rx_bytes` -
$if_hash{$if}[0];
$tx_sum += `cat /sys/class/net/$if/statistics/tx_bytes` -
$if_hash{$if}[1];
}
$avgrec = $rx_sum / $CHECKPERIOD;
$avgtrans = $tx_sum / $CHECKPERIOD;
On 3/24/21 4:13 PM, Harry Mangalam wrote:
> parsyncfp might be thought of as nothing more than a massive hack of
rsync,
> so more hacks would not be out of line.
> I've more or less hacked in the changes required to add multiple
interfaces
> in the (multihost) version that I'm working on now (tho the 2 are
> incompatible) and they should be ready to test within a week. I'll send
> them out to you when they are, prior to sending it to github.
>
> Monitoring the bandwidth isnt so much the problem - pfp already does that
> with perfquery (on IB) and other tickles, but organizing the returned
> output might be tricky. I'll play around with it and see what blows up in
> my face ;)
> hjm
>
> On Wed, Mar 24, 2021 at 12:52 PM Chris Rapier ***@***.***>
> wrote:
>
> > So I talked to my coworker and he basically hacked getavgnetbw and used
> > a wildcard to use ib* instead of ${NETIF}. That's dirty hack though. We
> > talked about it a bit and it's hard because the interface nomenclature
> > doesn't necessarily work with that outside of infiniband interfaces.
> > Unless you specifically ask the user to enter a regex or wildcard
> > solution that would work.
> >
> > However, if that can be solved it should be possible to just loop
> > through the results of the cat and aggregate the values of
> > /sys/class/net/[ifname]*/statistics/rx_bytes. Or even
> > /sys/class/net/[ifname]*/statistics/rx_bytes | paste -sd+ - | bc would
> > give you the total across all interfaces in a single command.
> >
> > He also said that if you make the --interface able to take a list of
> > interfaces you could just loop over that list. Which would be require
> > collecting the values in an array or hash. Which isn't awful as long as
> > you report aggregate values and not per interface. If that sounds like
> > it might work let me know and I'll see about getting you a patch. Might
> > take some time as I'm floored with other work right now.
> >
> > Chris
> >
> >
> > On 3/24/21 2:24 PM, Harry Mangalam wrote:
> > > so you want pfp to handle the detection and use of the rsync flag:
> > > --address=ADDRESS bind address for outgoing socket to daemon
> > >
> > > This would be on the same host, right?
> > >
> > > So when each new chunk is started up, it chooses its interface from
the
> > > list of available?/given? ones? I'd guess you'd probably want to
> specify
> > > the list so as not to bugger those that are being used for other
> > purposes.
> > > So something like:
> > >
> > > --roundrobin="192.168.0.4,
> > >
> >
>
192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"
> > >
> > > If not that, let me know what you're thinking.. The above doesn't
sound
> > > too hard. Just a slight mod of the flags to each rsync... altho I
could
> > > easily be missing something... Tho monitoring all those interfaces
> would
> > be
> > > a nightmare...
> > >
> > > hjm
> > >
> > >
> > >
> > > On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier ***@***.***>
> > > wrote:
> > >
> > > > Really, I'm just thinking about there being a list of interfaces
and
> > > > each rsync picking an interface from that list when it's
> instantiated.
> > > > Nothing on a byte level. Basically, we have a situation in our
> > > > datacenter where we have a host with 8x100Gb interfaces. All/most
of
> > the
> > > > interfaces are in the same CIDR block. So we were thinking that
> having
> > > > parsyncfp cycle through the interfaces would help with throughput
> > > > (assuming that the storage can keep up).
> > > >
> > > > Also, I want to say how helpful parsyncfp has been. We just moved
> > around
> > > > 3 or 4 PB of data from our old storage system to a new one.
> Lustre (of
> > > > course) was the big drag but parsyncfp worked great. I actually
> built a
> > > > front end to it so that users could easily build the parsyncfp
> command
> > > > and submit them as slurm jobs. I had to modify parsyncfp a bit -
some
> > of
> > > > those changes might be useful so I'll submit a pull request once
> I get
> > > > my version merged with HEAD.
> > > >
> > > > On 3/24/21 11:36 AM, Harry Mangalam wrote:
> > > > > And if it's really a round-robin use, how would you want the
> bytes to
> > > > > actually be allocated? I'm trying to think of a way that an
> actual RR
> > > > send
> > > > > would work, especially with the routing.. Can you elaborate? Or
> give
> > me
> > > > an
> > > > > example?
> > > > > h
> > > > >
> > > > > On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
> > > > > wrote:
> > > > >
> > > > > > It seems that parsyncfp won't make use of multiple interfaces
> on a
> > > > > > multihomed host. Before I start looking at doing myself I was
> > > > > wondering if
> > > > > > that was on your radar or if that functionality already
> exists and
> > > > > I'm just
> > > > > > missing it.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > —
> > > > > > You are receiving this because you are subscribed to this
thread.
> > > > > > Reply to this email directly, view it on GitHub
> > > > > > <#40>, or
> > unsubscribe
> > > > > >
> > > > > <
> > > >
> > >
> >
>
https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ
> > > > >
> > > > > > .
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Harry Mangalam
> > > > >
> > > > > —
> > > > > You are receiving this because you authored the thread.
> > > > > Reply to this email directly, view it on GitHub
> > > > > <
> > > >
> > >
> #40 (comment)
> > >,
> > > >
> > > > > or unsubscribe
> > > > > <
> > > >
> > >
> >
>
https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ
> > > > >.
> > > > >
> > > >
> > > > —
> > > > You are receiving this because you commented.
> > > > Reply to this email directly, view it on GitHub
> > > >
> > > <
> >
> #40 (comment)
>,
> > > > or unsubscribe
> > > >
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ
> > >
> > > > .
> > > >
> > >
> > >
> > > --
> > >
> > > Harry Mangalam
> > >
> > > —
> > > You are receiving this because you authored the thread.
> > > Reply to this email directly, view it on GitHub
> > > <
> >
> #40 (comment)
>,
> >
> > > or unsubscribe
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ
> > >.
> > >
> >
> > —
> > You are receiving this because you commented.
> > Reply to this email directly, view it on GitHub
> >
> <
#40 (comment)>,
> > or unsubscribe
> >
> <
https://github.com/notifications/unsubscribe-auth/AASF3YZFWNDZ6KZWUM6U2Y3TFI7G3ANCNFSM4ZXPJEHQ
>
> > .
> >
>
>
> --
>
> Harry Mangalam
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <
#40 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAKL66CBFBCVSI6KWMWLKADTFJBUZANCNFSM4ZXPJEHQ
>.
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AASF3Y4F5R64M77WWEHFFKTTFJCKPANCNFSM4ZXPJEHQ>
.
--
Harry Mangalam
Hi Chris,
I just put together this interface and it does emit the correct rsync
command, but with my dual-homed system (albeit only a 2wifi setup), it
really only uses one interface, regardless of the specified one.
My route shows:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
default router.asus.com 0.0.0.0 UG 601 0 0
wlx9cefd5fb0bb5
default router.asus.com 0.0.0.0 UG 602 0 0
wlp3s0
link-local 0.0.0.0 255.255.0.0 U 1000 0 0
wlp3s0
192.168.1.0 0.0.0.0 255.255.255.0 U 601 0 0
wlx9cefd5fb0bb5
192.168.1.0 0.0.0.0 255.255.255.0 U 602 0 0
wlp3s0
and the alternating rsync commands generated from pfp are:
rsync --address=192.168.1.223 --bwlimit=1000000 -a -s
--log-file=/home/hjm/.parsyncfp/rsync-logfile-14.34.52_2021-03-25_16
--files-from=/home/hjm/.parsyncfp/fpcache/f.16 '/home/hjm' bridgi
t:/home/hjm/test
and
rsync --address=192.168.1.186 --bwlimit=1000000 -a -s
--log-file=/home/hjm/.parsyncfp/rsync-logfile-14.34.52_2021-03-25_17
--files-from=/home/hjm/.parsyncfp/fpcache/f.17 '/home/hjm'
bridgit:/home/hjm/test
But the byte streams show only data flowing on one. This is the case
whether the rsyncs are started from parsyncfp or via separate rsyncs
started from separate shells.
Have you been able to send rsync data out simultaneously on different
interfaces and if so, let me know how you did it so I can incorporate that
approach into pfp.
On Wed, Mar 24, 2021 at 11:23 AM Harry Mangalam ***@***.***>
wrote:
… so you want pfp to handle the detection and use of the rsync flag:
--address=ADDRESS bind address for outgoing socket to daemon
This would be on the same host, right?
So when each new chunk is started up, it chooses its interface from the
list of available?/given? ones? I'd guess you'd probably want to specify
the list so as not to bugger those that are being used for other purposes.
So something like:
--roundrobin="192.168.0.4,
192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78
"
If not that, let me know what you're thinking.. The above doesn't sound
too hard. Just a slight mod of the flags to each rsync... altho I could
easily be missing something... Tho monitoring all those interfaces would be
a nightmare...
hjm
On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier ***@***.***>
wrote:
> Really, I'm just thinking about there being a list of interfaces and
> each rsync picking an interface from that list when it's instantiated.
> Nothing on a byte level. Basically, we have a situation in our
> datacenter where we have a host with 8x100Gb interfaces. All/most of the
> interfaces are in the same CIDR block. So we were thinking that having
> parsyncfp cycle through the interfaces would help with throughput
> (assuming that the storage can keep up).
>
> Also, I want to say how helpful parsyncfp has been. We just moved around
> 3 or 4 PB of data from our old storage system to a new one. Lustre (of
> course) was the big drag but parsyncfp worked great. I actually built a
> front end to it so that users could easily build the parsyncfp command
> and submit them as slurm jobs. I had to modify parsyncfp a bit - some of
> those changes might be useful so I'll submit a pull request once I get
> my version merged with HEAD.
>
> On 3/24/21 11:36 AM, Harry Mangalam wrote:
> > And if it's really a round-robin use, how would you want the bytes to
> > actually be allocated? I'm trying to think of a way that an actual RR
> send
> > would work, especially with the routing.. Can you elaborate? Or give me
> an
> > example?
> > h
> >
> > On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
> > wrote:
> >
> > > It seems that parsyncfp won't make use of multiple interfaces on a
> > > multihomed host. Before I start looking at doing myself I was
> > wondering if
> > > that was on your radar or if that functionality already exists and
> > I'm just
> > > missing it.
> > >
> > > Thanks!
> > >
> > > —
> > > You are receiving this because you are subscribed to this thread.
> > > Reply to this email directly, view it on GitHub
> > > <#40>, or unsubscribe
> > >
> > <
> https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ
> >
> > > .
> > >
> >
> >
> > --
> >
> > Harry Mangalam
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub
> > <
> #40 (comment)>,
>
> > or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ
> >.
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#40 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ>
> .
>
--
Harry Mangalam
--
Harry Mangalam
Let me check. Because of working from home (the pandemic edition) I'm
not on site to do this myself.
…On 3/26/21 12:29 PM, Harry Mangalam wrote:
Hi Chris,
I just put together this interface and it does emit the correct rsync
command, but with my dual-homed system (albeit only a 2wifi setup), it
really only uses one interface, regardless of the specified one.
My route shows:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
default router.asus.com 0.0.0.0 UG 601 0 0
wlx9cefd5fb0bb5
default router.asus.com 0.0.0.0 UG 602 0 0
wlp3s0
link-local 0.0.0.0 255.255.0.0 U 1000 0 0
wlp3s0
192.168.1.0 0.0.0.0 255.255.255.0 U 601 0 0
wlx9cefd5fb0bb5
192.168.1.0 0.0.0.0 255.255.255.0 U 602 0 0
wlp3s0
and the alternating rsync commands generated from pfp are:
rsync --address=192.168.1.223 --bwlimit=1000000 -a -s
--log-file=/home/hjm/.parsyncfp/rsync-logfile-14.34.52_2021-03-25_16
--files-from=/home/hjm/.parsyncfp/fpcache/f.16 '/home/hjm' bridgi
t:/home/hjm/test
and
rsync --address=192.168.1.186 --bwlimit=1000000 -a -s
--log-file=/home/hjm/.parsyncfp/rsync-logfile-14.34.52_2021-03-25_17
--files-from=/home/hjm/.parsyncfp/fpcache/f.17 '/home/hjm'
bridgit:/home/hjm/test
But the byte streams show only data flowing on one. This is the case
whether the rsyncs are started from parsyncfp or via separate rsyncs
started from separate shells.
Have you been able to send rsync data out simultaneously on different
interfaces and if so, let me know how you did it so I can incorporate that
approach into pfp.
On Wed, Mar 24, 2021 at 11:23 AM Harry Mangalam ***@***.***>
wrote:
> so you want pfp to handle the detection and use of the rsync flag:
> --address=ADDRESS bind address for outgoing socket to daemon
>
> This would be on the same host, right?
>
> So when each new chunk is started up, it chooses its interface from the
> list of available?/given? ones? I'd guess you'd probably want to specify
> the list so as not to bugger those that are being used for other
purposes.
> So something like:
>
> --roundrobin="192.168.0.4,
>
192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78
> "
>
> If not that, let me know what you're thinking.. The above doesn't sound
> too hard. Just a slight mod of the flags to each rsync... altho I could
> easily be missing something... Tho monitoring all those interfaces
would be
> a nightmare...
>
> hjm
>
>
>
> On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier ***@***.***>
> wrote:
>
>> Really, I'm just thinking about there being a list of interfaces and
>> each rsync picking an interface from that list when it's instantiated.
>> Nothing on a byte level. Basically, we have a situation in our
>> datacenter where we have a host with 8x100Gb interfaces. All/most of the
>> interfaces are in the same CIDR block. So we were thinking that having
>> parsyncfp cycle through the interfaces would help with throughput
>> (assuming that the storage can keep up).
>>
>> Also, I want to say how helpful parsyncfp has been. We just moved around
>> 3 or 4 PB of data from our old storage system to a new one. Lustre (of
>> course) was the big drag but parsyncfp worked great. I actually built a
>> front end to it so that users could easily build the parsyncfp command
>> and submit them as slurm jobs. I had to modify parsyncfp a bit - some of
>> those changes might be useful so I'll submit a pull request once I get
>> my version merged with HEAD.
>>
>> On 3/24/21 11:36 AM, Harry Mangalam wrote:
>> > And if it's really a round-robin use, how would you want the bytes to
>> > actually be allocated? I'm trying to think of a way that an actual RR
>> send
>> > would work, especially with the routing.. Can you elaborate? Or
give me
>> an
>> > example?
>> > h
>> >
>> > On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier ***@***.***>
>> > wrote:
>> >
>> > > It seems that parsyncfp won't make use of multiple interfaces on a
>> > > multihomed host. Before I start looking at doing myself I was
>> > wondering if
>> > > that was on your radar or if that functionality already exists and
>> > I'm just
>> > > missing it.
>> > >
>> > > Thanks!
>> > >
>> > > —
>> > > You are receiving this because you are subscribed to this thread.
>> > > Reply to this email directly, view it on GitHub
>> > > <#40>, or unsubscribe
>> > >
>> > <
>>
https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ
>> >
>> > > .
>> > >
>> >
>> >
>> > --
>> >
>> > Harry Mangalam
>> >
>> > —
>> > You are receiving this because you authored the thread.
>> > Reply to this email directly, view it on GitHub
>> > <
>>
#40 (comment)>,
>>
>> > or unsubscribe
>> > <
>>
https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ
>> >.
>> >
>>
>> —
>> You are receiving this because you commented.
>> Reply to this email directly, view it on GitHub
>>
<#40 (comment)>,
>> or unsubscribe
>>
<https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ>
>> .
>>
>
>
> --
>
> Harry Mangalam
>
--
Harry Mangalam
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAKL66HJSSWW6VKKO6UIDGDTFSY7HANCNFSM4ZXPJEHQ>.
So I poked at things a bit and to be honest it looks like the --address option in rsync is kind of useless. However, using the ssh option -oBinadAddress=ip seems to work as expected (though I'm running into some performance issues which I think is entirely due to my config here at home). This means that the multiple NICs could be used but requires the inclusion of ssh options on the rsync line. Which obviously means that it won't work between mounted filesystems on the same host. So a specific solution instead of a general one. I think I might try to find out who is maintaining rsync and ask about the --address issue.
edit: Thinking about it I bet the --address is only for rsyncs daemons and clients. If you are using the ssh subsystem for transport it will just drop that option instead of passing it along.
Already posted to the rsync list. But feel free to add your voice.
I'll check into the ssh solution. That would be useful for many ppl.
…On Fri, Mar 26, 2021, 12:59 PM Chris Rapier ***@***.***> wrote:
So I poked at things a bit and to be honest it looks like the --address
option in rsync is kind of useless. However, using the ssh option
-oBinadAddress=ip seems to work as expected (though I'm running into some
performance issues which I think is entirely due to my config here at
home). This means that the multiple NICs could be used but requires the
inclusion of ssh options on the rsync line. Which obviously means that it
won't work between mounted filesystems on the same host. So a specific
solution instead of a general one. I think I might try to find out who is
maintaining rsync and ask about the --address issue.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AASF3Y2XWSRS4HNJ5F3RUILTFTRRNANCNFSM4ZXPJEHQ>
.
I'mot seeing the ssh option working here in concert with rsync: ie , from
different shells, running thru 2 wlan interfaces:
rsync --rsh="ssh -oBindAddress=192.168.1.223" -av nacs bridgit:~/test
and
rsync --rsh="ssh -oBindAddress=192.168.1.186" -av tacg bridgit:~/test
both streams are going thru a single interface. I suspect because the arp
table sees the 2 interfaces as 1 MAC.
It's odd that the ssh command itself works correctly - ssh will bind to the
appropriate interface and the appropriate bytes stream thru the right
interface, but asking rsync to use that ssh channel is failing.
What is the format of your rsync commanda that worked as expected, if not
similar to the above?
Harry
…On Fri, Mar 26, 2021 at 12:59 PM Chris Rapier ***@***.***> wrote:
So I poked at things a bit and to be honest it looks like the --address
option in rsync is kind of useless. However, using the ssh option
-oBinadAddress=ip seems to work as expected (though I'm running into some
performance issues which I think is entirely due to my config here at
home). This means that the multiple NICs could be used but requires the
inclusion of ssh options on the rsync line. Which obviously means that it
won't work between mounted filesystems on the same host. So a specific
solution instead of a general one. I think I might try to find out who is
maintaining rsync and ask about the --address issue.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#40 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AASF3Y2XWSRS4HNJ5F3RUILTFTRRNANCNFSM4ZXPJEHQ>
.
--
Harry Mangalam
A guy from the rsync list replied to my question about this. His
explanation is that if all the interfaces on your multi-homed system are on
the same network, then without explicit routing, one interface (the one
with the lowest Metric from the route command) will be used. My home net
was set up this way, hence the failure to replicate your experience. If the
interfaces are on different nets, then multiple interfaces could be used as
you describe. I'll send you a trial version of pfp that should work as you
describe if the interfaces are on different networks. I'm still working on
replicating such a setup where I can test this locally.
And if the interfaces are bonded to stripe data, then you shouldn't need to
do any of this, which I suspect you already know.
Harry
…On Fri, Mar 26, 2021 at 4:48 PM Harry Mangalam ***@***.***> wrote:
I'mot seeing the ssh option working here in concert with rsync: ie , from
different shells, running thru 2 wlan interfaces:
rsync --rsh="ssh -oBindAddress=192.168.1.223" -av nacs bridgit:~/test
and
rsync --rsh="ssh -oBindAddress=192.168.1.186" -av tacg bridgit:~/test
both streams are going thru a single interface. I suspect because the arp
table sees the 2 interfaces as 1 MAC.
It's odd that the ssh command itself works correctly - ssh will bind to
the appropriate interface and the appropriate bytes stream thru the right
interface, but asking rsync to use that ssh channel is failing.
What is the format of your rsync commanda that worked as expected, if not
similar to the above?
Harry
On Fri, Mar 26, 2021 at 12:59 PM Chris Rapier ***@***.***>
wrote:
> So I poked at things a bit and to be honest it looks like the --address
> option in rsync is kind of useless. However, using the ssh option
> -oBinadAddress=ip seems to work as expected (though I'm running into some
> performance issues which I think is entirely due to my config here at
> home). This means that the multiple NICs could be used but requires the
> inclusion of ssh options on the rsync line. Which obviously means that it
> won't work between mounted filesystems on the same host. So a specific
> solution instead of a general one. I think I might try to find out who is
> maintaining rsync and ask about the --address issue.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#40 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AASF3Y2XWSRS4HNJ5F3RUILTFTRRNANCNFSM4ZXPJEHQ>
> .
>
--
Harry Mangalam
--
Harry Mangalam
Hi Chris,
I have an interim roundrobin version ready to test. I don't have a setup
with multiple physical interfaces to multiple routes that I can modify, so
I can send you the version which has some editable choices for you to try.
I'll describe that in the email.
If you send me a direct email, I'll send that version. Once it seems to be
working, I'll shove it to github for others to try.
It has a lot of debug output enabled by default to make sure it's
roundrobining correctly, but is otherwise functional.
Harry
…On Mon, Mar 29, 2021 at 8:23 AM Harry Mangalam ***@***.***> wrote:
A guy from the rsync list replied to my question about this. His
explanation is that if all the interfaces on your multi-homed system are on
the same network, then without explicit routing, one interface (the one
with the lowest Metric from the route command) will be used. My home net
was set up this way, hence the failure to replicate your experience. If the
interfaces are on different nets, then multiple interfaces could be used as
you describe. I'll send you a trial version of pfp that should work as you
describe if the interfaces are on different networks. I'm still working
on replicating such a setup where I can test this locally.
And if the interfaces are bonded to stripe data, then you shouldn't need
to do any of this, which I suspect you already know.
Harry
On Fri, Mar 26, 2021 at 4:48 PM Harry Mangalam ***@***.***>
wrote:
> I'mot seeing the ssh option working here in concert with rsync: ie ,
> from different shells, running thru 2 wlan interfaces:
>
> rsync --rsh="ssh -oBindAddress=192.168.1.223" -av nacs bridgit:~/test
> and
> rsync --rsh="ssh -oBindAddress=192.168.1.186" -av tacg bridgit:~/test
>
> both streams are going thru a single interface. I suspect because the
> arp table sees the 2 interfaces as 1 MAC.
>
> It's odd that the ssh command itself works correctly - ssh will bind to
> the appropriate interface and the appropriate bytes stream thru the right
> interface, but asking rsync to use that ssh channel is failing.
>
> What is the format of your rsync commanda that worked as expected, if not
> similar to the above?
>
> Harry
>
>
> On Fri, Mar 26, 2021 at 12:59 PM Chris Rapier ***@***.***>
> wrote:
>
>> So I poked at things a bit and to be honest it looks like the --address
>> option in rsync is kind of useless. However, using the ssh option
>> -oBinadAddress=ip seems to work as expected (though I'm running into some
>> performance issues which I think is entirely due to my config here at
>> home). This means that the multiple NICs could be used but requires the
>> inclusion of ssh options on the rsync line. Which obviously means that it
>> won't work between mounted filesystems on the same host. So a specific
>> solution instead of a general one. I think I might try to find out who is
>> maintaining rsync and ask about the --address issue.
>>
>> —
>> You are receiving this because you commented.
>> Reply to this email directly, view it on GitHub
>> <#40 (comment)>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AASF3Y2XWSRS4HNJ5F3RUILTFTRRNANCNFSM4ZXPJEHQ>
>> .
>>
>
>
> --
>
> Harry Mangalam
>
--
Harry Mangalam
--
Harry Mangalam