Received and size are the same
locojohn opened this issue · comments
upload progress module reports the same values for received and size. Contrary to a bug report #42 this bug is present even when not using http2 connections.
Could something be please done about it?
Best regards,
Andrejs
nginx -V
nginx version: nginx/1.11.4
built with OpenSSL 1.0.2h 3 May 2016
TLS SNI support enabled
configure arguments: --prefix=/usr --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --with-cc-opt=-I/usr/include --with-ld-opt=-L/usr/lib64 --http-log-path=/var/log/nginx/access_log --http-client-body-temp-path=/var/lib/nginx/tmp/client --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --with-file-aio --with-debug --with-http_v2_module --with-ipv6 --with-pcre --with-threads --without-http_auth_basic_module --without-http_geo_module --without-http_limit_conn_module --without-http_memcached_module --without-http_referer_module --without-http_scgi_module --without-http_split_clients_module --without-http_userid_module --without-http_uwsgi_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module --with-http_mp4_module --with-http_perl_module --with-http_stub_status_module --with-http_realip_module --add-module=external_module/taobao-nginx-http-concat-f3b52a5 --add-module=external_module/nginx-upload-progress-module-0.9.1 --add-module=external_module/headers-more-nginx-module-0.29 --with-http_ssl_module --without-stream_upstream_hash_module --without-stream_upstream_least_conn_module --without-stream_upstream_zone_module --without-stream_upstream_hash_module --without-stream_upstream_least_conn_module --without-stream_upstream_zone_module --without-stream_upstream_hash_module --without-stream_upstream_least_conn_module --without-stream_upstream_zone_module --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --user=nginx --group=nginx
nginx.conf
user nginx www;
worker_processes auto;
worker_cpu_affinity auto;
pcre_jit on;
worker_rlimit_core 500M;
working_directory /var/tmp/nginx;
error_log /var/log/nginx/error_log info;
events {
worker_connections 10000;
use epoll;
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
limit_req_zone $binary_remote_addr zone=zone-requests:10m rate=1r/s;
limit_req_log_level warn;
client_header_timeout 10s;
client_body_timeout 10s;
send_timeout 10s;
client_max_body_size 700m;
connection_pool_size 256;
client_body_buffer_size 16k;
client_header_buffer_size 8k;
large_client_header_buffers 4 16k;
request_pool_size 4k;
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 4096 4k;
open_file_cache off;
open_log_file_cache max=1000 inactive=60s valid=1m;
geoip_country /usr/share/GeoIP/GeoLiteCountry.dat; # the country IP data
geoip_city /usr/share/GeoIP/GeoLiteCity.dat; # the city IP database
gzip on;
gzip_min_length 1100;
gzip_buffers 4 8k;
gzip_types text/plain text/css application/json application/javascript a
output_buffers 1 32k;
postpone_output 1460;
aio on;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 75s;
keepalive_requests 100000;
keepalive_disable msie6;
ignore_invalid_headers on;
server_name_in_redirect on;
merge_slashes on;
fastcgi_pass_request_headers on;
fastcgi_intercept_errors off;
fastcgi_buffer_size 4k;
fastcgi_buffers 256 4k;
fastcgi_read_timeout 360; # xdebug needs larger value for fastcgi_
fastcgi_send_timeout 180;
fastcgi_store_access user:rw group:r;
fastcgi_force_ranges on;
proxy_intercept_errors off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 1h;
ssl_session_tickets on;
ssl_session_ticket_key ssl_session_ticket_file.key;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GC
ssl_prefer_server_ciphers on;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/ssl/certs/startssl-ca-bundle.crt;
ssl_certificate /etc/ssl/certs/1_mydomain.com_bundle.crt;
ssl_certificate_key /etc/ssl/private/1_mydomain.com_bundle.key;
index index.html;
charset utf-8;
# common headers
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
# upload progress
upload_progress uploads 1m;
upload_progress_content_type "application/json";
upload_progress_json_output;
concat on;
concat_max_files 40;
resolver 127.0.0.1 valid=300s;
resolver_timeout 10s;
# devel.mydomain:443
server {
listen 443 ssl;
server_name devel.mydomain;
access_log /var/log/nginx/mydomain.access_log main;
error_log /var/log/nginx/mydomain.error_log info;
root /opt/www/mydomain;
add_header Strict-Transport-Security "max-age=63072000;";
keepalive_timeout 70;
location / {
index index.php;
try_files $uri $uri/ @index;
}
location @index {
include fastcgi_params;
fastcgi_param PATH_INFO $uri;
fastcgi_param PATH_TRANSLATED $document_root$uri;
fastcgi_param SCRIPT_FILENAME $document_root/index.php;
fastcgi_pass unix:/run/php-fpm/3508-php70.sock;
track_uploads uploads 30s;
}
location = /upload-progress {
access_log off;
postpone_output 0;
report_uploads uploads;
}
}
}
Can you post or link to a debug log where you are doing an upload and send a few probes?
Most of the reports I got about this issue (in non-http2 case) were mostly because people were either uploading way faster than they send probes (so the probes arrives after the upload is finished, which can happen on localhost for instance) or were reusing upload ids.
Log attached.
@locojohn thanks for the log, unfortunately it doesn't show the uploads itself, only the progress report probes requests.
Can you redo the test, and capture the log from the moment nginx receives the initial upload request up to the subsequent report probes?
Thanks!
@locojohn you should have somewhere a log entry starting with trackuploads id found, this is where the upload is registered in the module.
Sorry, Brice, more detailed debug log attached. I was uploading a 145 MB mp3 mix to make sure the file isn't too small.
It looks like as though it still uses http2 connections, which is weird because I deliberately changed the line in config to:
listen 443 ssl;
Anyway, maybe the debug log helps you to identify an issue with http2 connections as well. Hopefully, you can reproduce this issue locally.
Yes, you're doing the upload as HTTP/2.0:
2016/08/18 15:31:38 [debug] 17722#17722: *1 http2 http request line: "POST /icm/?p=incident&WorkgroupID=12&IncidentClassID=1&IncidentTypeID=231&CreatorGroupID=34&ajax=1&c=attach_files&X-Progress-ID=22e95e83d881c172dd177282225e77f9 HTTP/2.0"
I know why it doesn't work with http2/spdy, http2 is using a completely different receive code path than the regular http. The upload tracker is based on nginx internals, and it's not surprising that it doesn't work anymore on new features. It would take some time to find and add the workarounds I had to add for http to the http2 code path, and I never found the spare time to do this (of course Pull Requests are welcome :).
Closing as duplicate of #45.
Hello, we are currently hit by this issue as well. However i am not able to write some code for this. But my boss is willing to spend some money on this feature. Would it help? Or can you recommend some other solution to be able to utilize nginx + httpv2 + tracking progress of uploaded files? thank you