Timeouts upon uploading attachments

[Daniel Truemper]

Hi!

I have a problem when uploading attachments to CouchDB. I really am not sure if this is a trombi issue, probably it is related to tornado's AsyncHTTPClient or even pycurl/libcurl but I am kind of desperate here as I do not know where to dig further.

Problem is the following. I have a process running Tornado's eventloop and the AsyncHTTPClient. This process is sending and receiving messages to and from other 'workers'. Then the master is uploading stuff into a CouchDB. in about 90% of all cases nothing happens.

But, when I am uploading a file of size 1.3/1.4 MB to CouchDB I am getting this error in my logs:

httpclient.py:354 - Exception in callback <functools.partial object at 0x2e43ec0> - (<type 'exceptions.TypeError'>, TypeError('expected string or buffer',), <traceback object at 0x2e4ca70>)
Traceback (most recent call last):
    File "tornado-1.1.1-py2.6.egg/tornado/httpclient.py", line 349, in _finish
      time_info=time_info))
    File "tornado-1.1.1-py2.6.egg/tornado/stack_context.py", line 128, in wrapped
      callback(*args, **kwargs)
    File "trombi/client.py", line 480, in _really_callback
      data = json.loads(response.body)
    File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
      return _default_decoder.decode(s)
    File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

After some debugging and printing I discovered that pycurl returned a HTTP 599 timeout error. So I increased the default request_timeout from 20 to 60 seconds but to no avail, again the same problem...

Does anyone have a hint on this?

The only real issue with trombi at this point is the not so well error handling. The curl client is closed and you have to restart the whole process in order to get access to CouchDB again...

Best
Daniel

[Daniel Truemper]

A little update: I think I have narrowed down the problem to the filesize. I am not able to upload anything with more than 1000000 bytes. I am almost sure that this is a problem of the AsyncHTTPClient/pycurl interface.

So my main question for the time being remains: has anyone been able to upload an attachment with more than 1 MB?

[Jyrki Pulliainen]

I haven't hit limits with CouchDBs attachment uploading. What CouchDB do you use?

And apparently there's a bug in Document.attach, it does not handle any errors properly. I've added #11 to separate it from here, as they might be separate issues.

[Daniel Truemper]

I am using CouchDB 1.0.2 built from source using couch(io/one/base)'s build-couchdb script.

If I upload the attachment via Futon everything works, which is why I don't think it is a CouchDB problem...

[Jyrki Pulliainen]

I created a small script to test the problem: https://gist.github.com/834928

I tried with 1M and 2M files, both working ok. Can you tell more about your setup? Does your app involve uploading files to a tornado instance and sending them to trombi?

[Daniel Truemper]

Yes, thanks for all the effort. The script just works on my side, so the problem must come from some other part of my code (unfortunately :)).

Closing this and thank you again for your help!
Daniel