Socket send buffer limitation

[Frédéric MOHIER]

We (@ddurieux and I) noticed problems with Livestatus and Thruk for large requests. For instance, it is possible to "GET hosts" up to 100 hosts but not over it ... 101 hosts make Thruk hangs !

After some testing, we found that :

1/ The underlying system has a socket buffer suze limitation when sending data to remote host (eg. 32768 bytes).

2/ The livestatus broker is sending ouput data without taking care about effective data sent:

                                try:
                                    s.send(output)
                                    self.write_protocol('', output)
                                except Exception, e:
                                    pass

3/ All the output data are not sent and then Thruk remains waiting for missing data.

One should test how many bytes have been sent and then attempt to deliver the remaining data ... I will make a test on my configuration.

[Gabès Jean]

+1, we should try a while True like on this example:
https://docs.python.org/2/howto/sockets.html#using-a-socket

and os we don't need to change the buffer size :)

On Fri, Sep 19, 2014 at 11:36 AM, Frédéric MOHIER [e-mail]
wrote:

We (@ddurieux https://github.com/ddurieux and I) noticed problems with
Livestatus and Thruk for large requests. For instance, it is possible to
"GET hosts" up to 100 hosts but not over it ... 101 hosts make Thruk hangs !

After some testing, we found that :

1/ The underlying system has a socket buffer suze limitation when sending
data to remote host (eg. 32768 bytes).

2/ The livestatus broker is sending ouput data without taking care about
effective data sent:

                            try:
                                s.send(output)
                                self.write_protocol('', output)
                            except Exception, e:
                                pass

3/ All the output data are not sent and then Thruk remains waiting for
missing data.

One should test how many bytes have been sent and then attempt to deliver
the remaining data ... I will make a test on my configuration.

—
Reply to this email directly or view it on GitHub
#37.

[Frédéric MOHIER]

Maximum send buffer, on Free BSD:

   sysctl -a net.inet.tcp.sendspace

Maximum send buffer, on Linux Debian:

   cat /proc/sys/net/core/wmem_max

This comment may be helpful ... for me, it is 👍

[Frédéric MOHIER]

Test en cours ;-)

[Frédéric MOHIER]

Success !

@naparuba : for such a bug, you win a tuto about Python socket ;-) You owe me a beer and for @ddurieux it will be a full truck of Coca ... he lost his nerves on this problem :-)

I make some more tests and then I will commit ...

                                try:
                                    totalsent = 0
                                    while totalsent < len(output):
                                        sent = s.send(output[totalsent:])
                                        if sent == 0:
                                            raise RuntimeError("socket connection broken")
                                        self.write_protocol('', output[totalsent:], sent)
                                        totalsent = totalsent + sent

                                except Exception, e:
                                    if self.debug_queries:
                                        print "EXCEPTION: %s \n\n\n" % str(e)
                                    pass

[Gabès Jean]

Cool congrats :)

On Fri, Sep 19, 2014 at 12:33 PM, Frédéric MOHIER [e-mail]
wrote:

Success !

@naparuba https://github.com/naparuba : for such a bug, you win a tuto
about Python socket ;-) You owe me a beer and for @ddurieux
https://github.com/ddurieux it will be a full truck of Coca ... he lost
his nerves on this problem :-)

I make some more tests and then I will commit ...

                            try:
                                totalsent = 0
                                while totalsent < len(output):
                                    sent = s.send(output[totalsent:])
                                    if sent == 0:
                                        raise RuntimeError("socket connection broken")
                                    self.write_protocol('', output[totalsent:], sent)
                                    totalsent = totalsent + sent

                            except Exception, e:
                                if self.debug_queries:
                                    print "EXCEPTION: %s \n\n\n" % str(e)
                                pass

—
Reply to this email directly or view it on GitHub
#37 (comment)
.