Lost connection in longPoll

Question

Lost connection in longPoll

anusoft opened this issue 10 years ago · comments

This error sometimes occur every 10 minutes, but sometime connection last many hours.

Traceback (most recent call last):
File "echobot.py", line 30, in
for op in client.longPoll():
File "/usr/local/lib/python2.7/dist-packages/line-0.1.8-py2.7.egg/line/client.py", line 379, in longPoll
operations = self._fetchOperations(self.revision, count)
File "/usr/local/lib/python2.7/dist-packages/line-0.1.8-py2.7.egg/line/api.py", line 255, in _fetchOperations
return self._client.fetchOperations(revision, count)
File "/usr/local/lib/python2.7/dist-packages/curve-0.1.0-py2.7.egg/curve/CurveThrift.py", line 482, in fetchOperations
self.send_fetchOperations(localRev, count)
File "/usr/local/lib/python2.7/dist-packages/curve-0.1.0-py2.7.egg/curve/CurveThrift.py", line 492, in send_fetchOperations
self._oprot.trans.flush()
File "/usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7.egg/thrift/transport/THttpClient.py", line 106, in _f
result = f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7.egg/thrift/transport/THttpClient.py", line 139, in flush
self.__http.endheaders()
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 772, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

Taehoon Kim · Answer 1 · Thu Nov 06 2014 15:12:36 GMT+0800 (China Standard Time)

Actually, This is not a new issue because some people told about this but I couldn't find the answer for this issue yet..

Taehoon Kim · Answer 2 · Tue Mar 31 2015 13:06:22 GMT+0800 (China Standard Time)

b5397aa, problem solved.

anusoft · Answer 3 · Wed Apr 01 2015 02:05:56 GMT+0800 (China Standard Time)

I've still found issue after upgraded:

Traceback (most recent call last):
File "echobot.py", line 34, in
for op in client.longPoll():
File "/usr/local/lib/python2.7/dist-packages/line-0.4.1-py2.7.egg/line/client.py", line 504, in longPoll
operations = self._fetchOperations(self.revision, count)
File "/usr/local/lib/python2.7/dist-packages/line-0.4.1-py2.7.egg/line/api.py", line 308, in _fetchOperations
return self._client.fetchOperations(revision, count)
File "/usr/local/lib/python2.7/dist-packages/curve-0.1.0-py2.7.egg/curve/CurveThrift.py", line 482, in fetchOperations
self.send_fetchOperations(localRev, count)
File "/usr/local/lib/python2.7/dist-packages/curve-0.1.0-py2.7.egg/curve/CurveThrift.py", line 492, in send_fetchOperations
self._oprot.trans.flush()
File "/usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7.egg/thrift/transport/THttpClient.py", line 106, in _f
result = f(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7.egg/thrift/transport/THttpClient.py", line 139, in flush
self.__http.endheaders()
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 772, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

Taehoon Kim · Answer 4 · Wed Apr 01 2015 10:40:45 GMT+0800 (China Standard Time)

I think this is not the problem of expiration of authToken. I guess Name or service not known error can be a problem of internet. How about making a exception around the longPoll and bypass this error?

anusoft · Answer 5 · Thu Apr 02 2015 03:05:51 GMT+0800 (China Standard Time)

There're other few error, I'll try to collect here and make exceptions for retrying.

But there're sometimes bots just freeze without disconnect and not responding, that is harder to fix and I think there should be some NOOP or PING to probe if connection is alive too.

anusoft · Answer 6 · Thu Apr 02 2015 15:54:05 GMT+0800 (China Standard Time)

I attempted to reconnect when socket error but still not working, any advise?

        if self._check_auth():
            """Check is there any operations from LINE server"""
            OT = OperationType

            try:
                operations = self._fetchOperations(self.revision, count)
            except SocketError as serr:
                print "Socket Error serr.errno %s" % serr.errno

                self.login()
                self.ready()

When I tried to call self.login(), self.read() it's show valid authToken but not able to continue

LoginResult(authToken='XXX, verifier=None, type=1, pinCode=None, certificate=None)
'XXX'
[*] DUMMY
Operation(status=None, checkSum=None, param3=None, param2=None, param1=None, createdTime=1427958741545, message=None, reqSeq=0, type=48, revision=747022)

Taehoon Kim · Answer 7 · Thu Apr 02 2015 16:31:25 GMT+0800 (China Standard Time)

I think you'd better use

client.updateAuthToken()

when the exception is occurred. It is a method that refresh authToken without PinCode authentication.

anusoft · Answer 8 · Thu Apr 02 2015 17:03:42 GMT+0800 (China Standard Time)

When the connection freezes, I've tried to check with netstats found that the connection is still alive, but no data receiving

# netstat -anp | grep 13114
tcp        1      0 172.16.1.1:59796        203.104.142.1:80        CLOSE_WAIT  13114/python    
tcp        0      0 172.16.1.1:60061        203.104.142.1:80        ESTABLISHED 13114/python    

# strace -p 13114
Process 13114 attached
recvfrom(4,

What method do you suggest for periodically check if connection is alive?

Edited: I tried to create work around for freezing bots by getProfile every minute.

t1 = time.time()
t2 = time.time()

while True:
    op_list = []

    if  (time.time() - t1) >= 60:
        print "Idle 60 Second"
        t1 = time.time()
        client.getProfile()

    if  (time.time() - t2) >= 7000:
        print "Idle 7000 Second"
        t2 = time.time()
        client.updateAuthToken()

Edited 2:

self.login() + self.ready() won't work wells when socket error, it will cause more socket error or another error. I decided to quit the program then rerun it again when socket error using shell

while [ true ]; do python echobot.py; sleep 10; done

Also getProfile() every minute and updateAuthToken() every almost two hours seem does the job well, I can run bots continuously more than 24 hours now without getting any socket error.

Edited 3:

It still freeze from time to times so I'm using threading to check if it's still response.

import os
import threading

def DEAD():
    print 'DEAD'
    os._exit(1)    

tt = threading.Timer(65.0, DEAD)
tt.start()

while True:
    op_list = []

    if  (time.time() - t1) >= 60:
        print time.strftime("%Y-%m-%d %H:%M:%S"), "Idle 60 Second"
        t1 = time.time()
        client.getProfile()

        tt.cancel()
        tt = threading.Timer(65.0, DEAD)
        tt.start()

anusoft · Answer 9 · Tue Jul 14 2015 01:50:21 GMT+0800 (China Standard Time)

#25 seem to be also fixed this.