Getting 500 Internal Server Error from nginx when upload progress is enabled

[anvio]

I am constantly getting a 500 Internal Server Error from nginx when I try to upload a file to the server.
There is nothing in the error log that corresponds to this error. This happens on a server that is under heavy load. There are several hundreds of users on the website at the same time.

I am using the following configuration:
Nginx 1.1.19 incl. nginx-extras with upload progress module (Debian backports) but I also tried to compile everything myself from the latest sources (0.8.4 and 0.9 from Github).

My nginx.conf:

user www-data;
worker_processes 1;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
multi_accept off;
}

http {
include /etc/nginx/mime.types;

access_log  /var/log/nginx/access.log;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;
tcp_nodelay        on;

gzip  on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";

upload_progress     uploadtracker 1m;

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

}

My site configuration file:

server {
listen 80; ## listen for ipv4
listen [::]:80 default ipv6only=on; ## listen for ipv6
error_log /var/log/nginx/error.site.log;

    server_name  *.domain.tld;

    location /static/ {
            alias /path/to/static/;
    }

    location ~* /upload_progress {
            upload_progress_json_output;
            report_uploads uploadtracker;
    }

    location / {
            access_log      off;
            proxy_pass http://localhost:8080;
            proxy_set_header X-Real-IP  $remote_addr;
            client_max_body_size 1000M;
            track_uploads uploadtracker 6s;
    }

}

Way to reproduce the problem in my case:

1. upload file (size doesn't matter)
2. requesting upload progress from nginx. sometimes at the beginning the upload progress modules works for a few seconds and shows the upload progress but suddenly it returns a JSON object containing {state: "error", status: 500}.
3. The URL where the file was sent to returns a 500 Internal Server Error message from nginx.
4. After that all requests for upload progress get the JSON object mentioned in step 2.

Maybe it has something to do with memory allocation. I don't know. But it does not seem to work under heavy load.

Any ideas? Maybe how I can make upload progress module errors appear in the nginx error.log?

[Brice Figureau]

There are a handful of place in the module where we return a NGX_HTTP_INTERNAL_SERVER_ERROR (ie a 500 error code). Most of them are safeguards from allocation failures that could happen, and one logs an error at info level.
If your nginx has an error_log directive that logs at info, do you see the "upload_progress: tracking already registered id" error message?

If not, then the best is to activate the debug log for your nginx server by compiling it (whatever version it is) with --with-debug. Then use debug_connection ; to limit the debug error log to only your IP address (which might help if your server is under high load).

Next, do the upload until it fails, and send me the error log so that I can analyze it.

A possibility that comes to mind is that your server is seeing too many uploads and that doesn't fit in the 1 MiB you selected for the "uploadtracker" upload_progress directive. You might try to make this 10 or 20MiB and see if that helps.

I can also produce a patch that adds more logging in case of NGX_HTTP_INTERNAL_SERVER_ERROR to better find the problem (actually you can modify your very own version of the module with error logs before every return NGX_HTTP_INTERNAL_SERVER_ERROR).

[anvio]

Thank you for the very quick response!
I just enabled the error log at info level and I see the error message
you've mentioned.

2012/05/16 16:05:26 [info] 467#0: *523702 upload_progress: tracking
already registered id: d2938165-07d5-4f14-94a9-fdcdefc77b9f, client:
xxx.xxx.xxx.xxx, server: *.domain.tld, request:
"POST /uploadafile?X-Progress-ID=d2938165-07d5-4f14-94a9-fdcdefc77b9f
HTTP/1.1", host: "www.domain.tld", referrer: "http://www.domain.tld/"

Does this already help? I've also tried to increase the storage in
upload_progress before but it didn't work either.

[anvio]

I just recognized that there is indeed the same uuid used for many different visitors. I have to check that.

[anvio]

It is fixed now! It was my fault. The uuid was shared over multiple Apache instances.
Thank you very much for your time and help!