Sish server crashes when transferring big files

Question

Sish server crashes when transferring big files

manprint opened this issue 2 years ago · comments

Hi AntonioMika, congratulations for the excellent sish tool. He saved me on several occasions.
I am trying to forward a webdav server but when I try to transfer large files (download and upload) (~1GB size) Sish server crashes (deployed with docker and apache in front) without any information in the logs (also tried in debug mode). The webdav server works fine with large files without sish.

The command I use for the server is the following:

docker run -itd --name sish \
	-v $(pwd)/keys:/keys \
	-v $(pwd)/pubkeys:/pubkeys \
	--restart=always \
	-p 2222:2222 \
	-p 4443:4443 \
	-p 8765:80 \
	antoniomika/sish:2.7.0 \
	--ssh-address=:2222 \
	--http-address=:80 \
	--https=true \
	--authentication-keys-directory=/pubkeys \
	--private-keys-directory=/keys \
        --tcp-aliases \
        --bind-random-aliases=false \
	--bind-random-ports=false \
	--bind-random-subdomains=false \
	--authentication-password= \
	--domain=mysishserver.com \
	--idle-connection=true \
	--idle-connection-timeout=360s \
        --ping-client-timeout=360s \
	--cleanup-unauthed-timeout=360s \
        --cleanup-unbound \
	--cleanup-unbound-timeout=360s \
	--admin-console \
	--admin-console-token=my-secret-token \
	--verify-ssl=false

The apache configuration is as follows:

<VirtualHost *:80>

	ServerName mysishserver.com
	ServerAlias *.mysishserver.com

	#RewriteEngine On
	#RewriteCond %{HTTPS} !=on
	#RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

	# Websocket Support
	RewriteEngine on
	RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
	RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
	RewriteRule .* ws://172.30.0.72:8765%{REQUEST_URI} [P,QSA,L]

	ProxyPreserveHost On

	ProxyPass        / http://172.30.0.72:8765/
	ProxyPassReverse / http://172.30.0.72:8765/

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

<VirtualHost *:443>

	ServerName mysishserver.com
	ServerAlias *.mysishserver.com

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	SSLEngine on

	SSLCertificateFile      /home/ubuntu/docker_services/sish/ssl-keys/tls.crt
	SSLCertificateKeyFile   /home/ubuntu/docker_services/sish/ssl-keys/tls.key

	ProxyPreserveHost On
	RequestHeader set X-Forwarded-Proto "https"

	# Websocket Support
	RewriteEngine on
	RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
	RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
	RewriteRule .* ws://172.30.0.72:8765%{REQUEST_URI} [P,QSA,L]

	ProxyPass        / http://172.30.0.72:8765/
	ProxyPassReverse / http://172.30.0.72:8765/

	<FilesMatch "\.(cgi|shtml|phtml|php)$">
					SSLOptions +StdEnvVars
	</FilesMatch>
	<Directory /usr/lib/cgi-bin>
					SSLOptions +StdEnvVars
	</Directory>

</VirtualHost>

The connection is started with:

ssh -o 'PubkeyAcceptedKeyTypes +ssh-rsa' -p 2222 -R webdav:80:localhost:80 mysishserver.com

Do I make any errors in the configuration?
Thank you

Antonio Mika · Answer 1 · Thu Oct 20 2022 05:36:10 GMT+0800 (China Standard Time)

To me your configuration looks fine. Can you run sish with --debug=true and send the logs here?

Manprint · Answer 2 · Thu Oct 20 2022 05:52:18 GMT+0800 (China Standard Time)

This is the log.
sish.log

Thanks

Antonio Mika · Answer 3 · Thu Oct 20 2022 05:56:31 GMT+0800 (China Standard Time)

Hey @manprint,

Don't see any crashes there unfortunately. Does the sish server itself crash or just the ssh client? If the sish server itself crashes, mind giving me the output of docker inspect <sish container id>. Be cautious with this output just in case there are environment secrets you may have added to the container.

Manprint · Answer 4 · Thu Oct 20 2022 06:04:04 GMT+0800 (China Standard Time)

The server crash. The docker container restarts.

container-inspect.log

Thank you

PS: This is the webdav server: https://hub.docker.com/r/bytemark/webdav/

Antonio Mika · Answer 5 · Thu Oct 20 2022 06:27:53 GMT+0800 (China Standard Time)

Hrm I don't see any issues here. How are you grabbing the longs? If you're doing console redirection, you might need to include stderr as well.

Manprint · Answer 6 · Thu Oct 20 2022 06:38:08 GMT+0800 (China Standard Time)

I got the sish server logs directly from docker logs -f sish without any redirection
Tomorrow I can try to transfer files of different sizes.
So it is possible to see if there is a size limit or something similar.
Thank you very much

Manprint · Answer 7 · Thu Oct 20 2022 18:02:56 GMT+0800 (China Standard Time)

Hi Antoniomika, I have made some tests with different files of different sizes. These are the results:

-rw-rw-r-- 1 user user   50M ott 20 11:34 test50m.txt -> OK
-rw-rw-r-- 1 user user  100M ott 20 11:34 test100m.txt -> OK
-rw-rw-r-- 1 user user  200M ott 20 11:34 test200m.txt -> OK
-rw-rw-r-- 1 user user  300M ott 20 11:34 test300m.txt -> FAIL (sish crash)
-rw-rw-r-- 1 user user  400M ott 20 11:35 test400m.txt -> FAIL (sish crash)

I noticed that as the file size grows, the RAM memory allocated by Sish grows.
With a 300MB file the container's ram rises to 2GB and the server crashes (the container is killed by the operating system for saturation of the RAM?)

dcb821935b14  sish    0.23%    2.136GiB / 3.786GiB    0.96%    7.11kB / 4.79kB     16.2MB / 0B         9

I think this is the issue.
What do you think about it?

The VM on which I run the server is an AWS EC2 with 2 cores and 4 GB Ram.
Thanks!

Antonio Mika · Answer 8 · Thu Oct 20 2022 19:58:50 GMT+0800 (China Standard Time)

That's what I thought it could be, and which is why I wanted the inspect output as the exit code is sometimes included there if caught quick enough. Oxy (the lib we are using) might buffer output. In this case, would a SNI proxy endpoint or TCP endpoint suffice (or at least be used to test this issue)? From what I recall of how I implemented those, it should be streaming instead of buffer based.

Manprint · Answer 9 · Thu Oct 20 2022 23:06:45 GMT+0800 (China Standard Time)

OK, tonight I'll try a tcp tunnel instead of http forwarding. I will let you know.
Thanks.

Manprint · Answer 10 · Fri Oct 21 2022 01:13:22 GMT+0800 (China Standard Time)

Hi, I have run tests with the TCP tunnel. With this method, sish works well with large files.

ssh -o 'PubkeyAcceptedKeyTypes +ssh-rsa' -p 2222 -R 81:localhost:80 apps.internaltest.tk

response:

Starting SSH Forwarding service for tcp:81. Forwarded connections can be accessed via the following methods:
TCP: apps.internaltest.tk:81

Below is the result of the tests:

-rw-rw-r-- 1 user user   50M ott 20 11:34 test50m.txt -> OK
-rw-rw-r-- 1 user user  100M ott 20 11:34 test100m.txt -> OK
-rw-rw-r-- 1 user user  200M ott 20 11:34 test200m.txt -> OK
-rw-rw-r-- 1 user user  300M ott 20 11:34 test300m.txt -> OK
-rw-rw-r-- 1 user user  500M ott 20 18:46 test500m.txt -> OK
-rw-rw-r-- 1 user user  500M ott 20 18:47 test700m.txt -> OK
-rw-rw-r-- 1 user user  900M ott 20 18:47 test900m.txt -> OK
-rw-rw-r-- 1 user user 1000M ott 20 18:27 test1000m.txt -> OK

The crash problem occurs when http and https forwarding are done.
Probably because of the buffering issue.

Is there a way to use streaming instead of buffering in HTTP / HTTPS farwarding?

If you need to do other tests please ask me. (I am an Italian Devops Engineer).
I'm glad to help you develop and test your app.

Thanks a lot. Fabio

Antonio Mika · Answer 11 · Fri Oct 21 2022 01:44:28 GMT+0800 (China Standard Time)

Hey Fabio,

Thanks for checking, glad that works. I have a method for enabling streaming and I'm going to test that. I'll have a build you can test in about an hour!

Best,

Manprint · Answer 12 · Fri Oct 21 2022 03:47:08 GMT+0800 (China Standard Time)

Ok, thanks.
When it is ready I try if everything works without problems.
Good work and I wish you a good evening!
Fabio

Antonio Mika · Answer 13 · Fri Oct 21 2022 05:38:48 GMT+0800 (China Standard Time)

Hey @manprint!

Just pushed the PR and the image was created for testing, give it a go and make sure to set --service-console-max-content-length to something that works with your ram settings (or just 0 if you don't really use the service console for dumping request/response data).

antoniomika/sish:am-http-streaming-and-limit

Manprint · Answer 14 · Fri Oct 21 2022 06:04:47 GMT+0800 (China Standard Time)

Hi @antoniomika. Thank You Very Much.
Tomorrow I will do the file transfer tests again with HTTP / HTTPS forwarding and let you know if everything works without issues
I'll set the --service-console-max-content-length=0 flag.
Best,
Fabio

Manprint · Answer 15 · Fri Oct 21 2022 06:42:20 GMT+0800 (China Standard Time)

Hi Antonio Mika, I did a fly test with the image you posted.
When I try to login to my webdav server there is this error in the sish server.
The same error occurs when I try to upload or download via rclone webdav client
The flag is setted to --service-console-max-content-length=0
I am attaching the log:

sish_streming_http_log.txt

Best,
Fabio

Antonio Mika · Answer 16 · Fri Oct 21 2022 08:08:21 GMT+0800 (China Standard Time)

@manprint thanks for the quick test! I found the issue and fixed it. Feel free to repull the image when you can.

Manprint · Answer 17 · Fri Oct 21 2022 18:30:19 GMT+0800 (China Standard Time)

Hi @antoniomika.
I have tested the image you pushed and everything seems to be working fine.
Large file transfer via webdav works well. I have tested with 1GB and 2GB files (download and upload).
The memory of the sish server is stable and does not grow as before. I send you the docker stats during the transfer.

0d8e9809b182   sish   4.67%   30.91MiB / 3.786GiB   0.80%   1.45GB / 1.38GB   0B / 0B  9

My colleagues right now are using Sish via both HTTP / HTTPS and TCP forwarding and everything looks ok.
At this moment there are 3 connections open and everything works fine:

2022/10/21 - 10:17:00 | =======Start=========
2022/10/21 - 10:17:00 | ===Goroutines=====
2022/10/21 - 10:17:00 | 72
2022/10/21 - 10:17:00 | ===Listeners======
2022/10/21 - 10:17:00 | /tmp/91.51.32.153_52938_801169233557
2022/10/21 - 10:17:00 | /tmp/91.51.32.153_57476_801392789932
2022/10/21 - 10:17:00 | :2222
2022/10/21 - 10:17:00 | /tmp/91.51.32.153_37934_80703006580
2022/10/21 - 10:17:00 | ===Clients========
2022/10/21 - 10:17:00 | 91.51.32.153:37934 alpine [/tmp/91.51.32.153_37934_80703006580]
2022/10/21 - 10:17:00 | 91.51.32.153:57476 alpine [/tmp/91.51.32.153_57476_801392789932]
2022/10/21 - 10:17:00 | 91.51.32.153:52938 alpine [/tmp/91.51.32.153_52938_801169233557]
2022/10/21 - 10:17:00 | ===HTTP Listeners===
2022/10/21 - 10:17:00 | http://:@testpaypal.apps.internaltest.tk [91.51.32.153:37934]
2022/10/21 - 10:17:00 | http://:@my-sio-ws.apps.internaltest.tk [91.51.32.153:57476]
2022/10/21 - 10:17:00 | http://:@webdav.apps.internaltest.tk [91.51.32.153:52938]
2022/10/21 - 10:17:00 | ===TCP Aliases====
2022/10/21 - 10:17:00 | ===TCP Listeners====
2022/10/21 - 10:17:00 | ===Web Console Routes====
2022/10/21 - 10:17:00 | http://:@webdav.apps.internaltest.tk []
2022/10/21 - 10:17:00 | http://:@my-sio-ws.apps.internaltest.tk []
2022/10/21 - 10:17:00 | http://:@testpaypal.apps.internaltest.tk []
2022/10/21 - 10:17:00 | ===Web Console Tokens====
2022/10/21 - 10:17:00 | http://:@testpaypal.apps.internaltest.tk 5xBiOwOascJR1vR2KTdP
2022/10/21 - 10:17:00 | http://:@my-sio-ws.apps.internaltest.tk dekGFK1oZAykpWm3qlz9
2022/10/21 - 10:17:00 | http://:@webdav.apps.internaltest.tk uTFPQuzMNGYbUTecZhDD
2022/10/21 - 10:17:00 | ========End==========

I believe the issue is resolved and can be closed.

For now I will use the image you pushed (maybe don't delete it yet until you bring the fix to the main branch and do the main release)
These days I will continue to use sish. If I find any other issue I will inform you.

Thank you for your help.

Best,
Fabio

Antonio Mika · Answer 18 · Fri Oct 21 2022 23:47:07 GMT+0800 (China Standard Time)

Thanks for checking! Just merged and a latest release will be tagged and available in a few minutes!

I don't currently clean up old branch images, as their size are so small (thanks scratch images!) and I like being able to go back in time to see whether or not things broke between changes. Feel free to continue to use it until you can take one of the latest ones :)

Manprint · Answer 19 · Fri Oct 21 2022 23:49:04 GMT+0800 (China Standard Time)

Ok, Thank You Very Much! 👍 🥇
Fabio