nginx. stream cuts off after a certain point
opened this issue · comments
i am using a portable bin and set up nginx as a proxy
when trying to read an infinite stream from the pipe server it cuts off at a certain point
seq inf only return up to about 393936. sends for less then a second
receiver error curl: (18) transfer closed with outstanding read data remaining
sender error curl: (55) OpenSSL SSL_write: Broken pipe, errno 32
2022/02/25 23:16:12 [error] 433215#433215: *288 upstream timed out (110: Connection timed out) while reading upstream, client: 72.178.17.200, server: x.grimmygrom.com, request: "PUT /testing HTTP/1.1", upstream: "http://127.0.0.1:8080/testing", host: "x.grimmygrom.com"
2022/02/25 23:16:12 [error] 433215#433215: *290 upstream prematurely closed connection while reading upstream, client: 72.178.17.200, server: x.grimmygrom.com, request: "GET /testing HTTP/1.1", upstream: "http://127.0.0.1:8080/testing", host: "x.grimmygrom.com"
my nginx config is like this
server {
server_name x.grimmygrom.com;
client_max_body_size 0;
proxy_http_version 1.1; # default: 1.0, Streaming (chunked transfer encoding) requires at least 1.1.
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_pass http://127.0.0.1:8080/;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/x.grimmygrom.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/x.grimmygrom.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = x.grimmygrom.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name x.grimmygrom.com;
listen 80;
return 404; # managed by Certbot
}
video of me showing the problem
https://youtu.be/WyqEju096so
thank you
Thank you for the detail report. I will see.
What happens when using HTTP instead HTTPS?
Can you ssh-login to the server of x.grimmygrom.com and do curl localhost:8080?
localhost:8080 works fine
What about localhost:80?
that goes to nginx. and i get a 404
What about curl -k https://localhost:443/testing? The -k option ignore the certificate.
that goes to nginx and i get 404
i set pipe server has port 4321 for https
and when i do curl -k https://localhost:4321/testing
that freezes and i get no output
It's OK. I mean that could you try seq inf | curl -kT- https://localhost:4321/testing and curl -k https://localhost:4321/testing like the video?
that works just fine.
I'm not sure why HTTPS port is 4321. The nginx configuration says 443. Do you have another reverse proxy for x.grimmygrom.com?
Can you publish https://localhost:4321 directly for x.grimmygrom.com?
i don't have another reverse proxy.
when i try launching pipe server with port 443. it gives me a error and says
[2022-03-06T02:25:41.483] [ERROR] default - on uncaughtException Error: listen EADDRINUSE: address already in use :::443
i had pipe server https on 4321 for testing
what do you mean by publish?
I guess the error message address already in use :::443 means that the port 443 is already used for nginx.
that goes to nginx and i get 404
Is https://localhost:443 https://x.grimmygrom.com:443?
yes they are both the same
If they are the same, doing seq inf | curl -kT- https://localhost:443/testing and curl -k https://localhost:443/testing are the same result as the video? I don't know why you got 404 error.
dose not work. i get error code 405 Not Allowed
my bad its not the same
i think this is due to it needed to be named x.grimmygrom.com so that localhost:443 to work
OK. I understand. You can attach -H "Host: x.grimmygrom.com" to the curl commands. Could you try?
that gives me the same error as in this video
streaming stops
Good. I'd like to reproduce on local.
Do you have pv command? Could you install it?
i dont? i dont know how to use pv
In my opinion, Nginx is very good software, but it is not good fit to HTTP streaming.
Could you try to add pv -qL 100k between seq inf and curl -kT- like below?
seq inf | pv -qL 100k | curl -kT- ...That command limits transfer speed and it works on my local.
this seems to work and not die
Good!
I'd not like to limits transfer speed, but unfortunately I have only this solution for now.
Since Ngix has many options, some options may allow us do streaming well. Maybe it is better to ask Nginx.
thank you so much for the help.
this is very helpful.
Thanks. Your video on YouTube was very good for me to understand your situation.
@m2acgi How did you send the big file? e.g. curl -T yourbigfile.dat https:///.
If you use curl -T yourbigfile.dat ..., the content-length is specified without chunked encoding. If content-length exists, Nginx should be handle without any error.
Yes, curl -T. Is there any nginx reverse proxy doc for piping-server?
curl -T- and curl -T <your filepath> are different. Which did you use?
curl -T- creates chunked encoding without content-length and curl -T <you filepath> creates a requet with content-length. With content-length, nginx should handle without any error.
I use the curl -T <filepath> in my test. Thank u for ur explain.