Disconnects when sitting unused for some time

Question

Disconnects when sitting unused for some time

evil-dog opened this issue 6 years ago · comments

I'm getting disconnected when the SSH session goes unused for some amount of time. I can't determine a specific amount of time, or other conditions, but it seems to be when the connection sits unused for 3-5 min (this is anecdotal, I have not actually tested it). I've only seen this after the connection is idle for some amount of time, I've not seen this happen to an active connection.

Generally what will happen is:

I login to my system using secure shell extension through nassh-relay
I do some stuff through GNU Screen
I go away to either browse the web for something, or away from the computer altogether
I come back to the SSH session and start typing

The client will generally show the first 1 or 2 chars I type, then will hang for a sec and then disconnect. When I reconnect and restart GNU Screen the 1 or 2 chars above plus 1 or 2 more chars will be shown in the console.

So based on what I see as it disconnects and the error message from the client, it seems like something is going wrong on the server. Either with nassh-relay or with my nginx reverse proxy config.

The secure shell client gives me the following error message:

Bad packet length 286636368.
ssh_dispatch_run_fatal: Connection to UNKNOWN port -1: Connection corrupted
NaCl plugin exited with status code 255.

This is the stack trace output by nassh-relay:

03:24:31.964 [vert.x-eventloop-thread-0]      ERROR i.v.c.i.ContextImpl - Unhandled exception
java.lang.IllegalStateException: Response is closed
        at io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:548)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end0(HttpServerResponseImpl.java:401)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:319)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:308)
        at net.zyclonite.nassh.util.TransferObserver.update(TransferObserver.java:46)
        at java.base/java.util.Observable.notifyObservers(Observable.java:173)
        at java.base/java.util.Observable.notifyObservers(Observable.java:129)
        at net.zyclonite.nassh.util.TransferQueue.add(TransferQueue.java:36)
        at net.zyclonite.nassh.handler.ProxyHandler.lambda$connectTcpEndpoint$5(ProxyHandler.java:137)
        at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:384)
        at io.vertx.core.net.impl.NetSocketImpl.handleMessageReceived(NetSocketImpl.java:351)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:242)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:239)
        at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146)
        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
        at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195)
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:144)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:844)
03:24:31.965 [vert.x-eventloop-thread-0]      ERROR i.v.c.i.ContextImpl - Unhandled exception
java.lang.IllegalStateException: Response is closed
        at io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:548)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end0(HttpServerResponseImpl.java:401)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:319)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:308)
        at net.zyclonite.nassh.util.TransferObserver.update(TransferObserver.java:46)
        at java.base/java.util.Observable.notifyObservers(Observable.java:173)
        at java.base/java.util.Observable.notifyObservers(Observable.java:129)
        at net.zyclonite.nassh.util.TransferQueue.add(TransferQueue.java:36)
        at net.zyclonite.nassh.handler.ProxyHandler.lambda$connectTcpEndpoint$5(ProxyHandler.java:137)
        at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:384)
        at io.vertx.core.net.impl.NetSocketImpl.handleMessageReceived(NetSocketImpl.java:351)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:242)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:239)
        at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146)
        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
        at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195)
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:144)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:844)
03:24:31.965 [vert.x-eventloop-thread-0]      ERROR i.v.c.i.ContextImpl - Unhandled exception
java.lang.IllegalStateException: Response is closed
        at io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:548)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end0(HttpServerResponseImpl.java:401)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:319)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:308)
        at net.zyclonite.nassh.util.TransferObserver.update(TransferObserver.java:46)
        at java.base/java.util.Observable.notifyObservers(Observable.java:173)
        at java.base/java.util.Observable.notifyObservers(Observable.java:129)
        at net.zyclonite.nassh.util.TransferQueue.add(TransferQueue.java:36)
        at net.zyclonite.nassh.handler.ProxyHandler.lambda$connectTcpEndpoint$5(ProxyHandler.java:137)
        at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:384)
        at io.vertx.core.net.impl.NetSocketImpl.handleMessageReceived(NetSocketImpl.java:351)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:242)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:239)
        at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146)
        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
        at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195)
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:144)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:844)
03:24:31.966 [vert.x-eventloop-thread-0]      ERROR i.v.c.i.ContextImpl - Unhandled exception
java.lang.IllegalStateException: Response is closed
        at io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:548)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end0(HttpServerResponseImpl.java:401)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:319)
        at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:308)
        at net.zyclonite.nassh.util.TransferObserver.update(TransferObserver.java:46)
        at java.base/java.util.Observable.notifyObservers(Observable.java:173)
        at java.base/java.util.Observable.notifyObservers(Observable.java:129)
        at net.zyclonite.nassh.util.TransferQueue.add(TransferQueue.java:36)
        at net.zyclonite.nassh.handler.ProxyHandler.lambda$connectTcpEndpoint$5(ProxyHandler.java:137)
        at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:384)
        at io.vertx.core.net.impl.NetSocketImpl.handleMessageReceived(NetSocketImpl.java:351)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:242)
        at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:239)
        at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146)
        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
        at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195)
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:144)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:844)

Here is my nassh-relay config:

{
  "webservice": {
    "hostname": "0.0.0.0",
    "port": 8022
  },

  "application": {
    "authentication": false,
    "max-sessions": 100,
    "tcp-session-timeout": 1200,
    "auth-session-timeout": 600
  },

  "accesslist": [{
    "email": "<redacted>",
    "access": [
      {"network": "0.0.0.0/0"}
    ]
  }],

  "whitelist": [
    {"network": "192.168.0.0/16"}
  ],

  "blacklist": [
    {"network": "169.254.0.0/16"},
    {"network": "172.16.0.0/12"},
    {"network": "10.0.0.0/8"},
    {"network": "0.0.0.0/8"},
    {"network": "fe80::10"},
    {"network": "127.0.0.1"}
  ],

  "google-sso": {
    "title": "Not used",
    "client-id": "",
    "client-secret": ""
  }
}

My nginx config for this vhost:

server {
  listen [::]:8022 ssl http2 ipv6only=off;
  server_name ssh-relay.<redacted>;

  access_log /var/log/nginx/ssh_relay_ssl.access.log;
  error_log /var/log/nginx/ssh_relay_ssl.error.log;

  include configlets/ssl-with-client-cert.conf;

  set $ssh_relay_backend "http://ssh-relay:8022";

  location / {
    proxy_set_header Host $server_name:$server_port;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_max_temp_file_size 0;
    proxy_buffering off;
    proxy_pass $ssh_relay_backend;
  }

  location /connect {
    proxy_pass $ssh_relay_backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $server_name:$server_port;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 10m;
    proxy_set_header Host $server_name:$server_port;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_max_temp_file_size 0;
    proxy_buffering off;
  }
}

Lukas Prettenthaler · Answer 1 · Tue Sep 11 2018 16:20:46 GMT+0800 (China Standard Time)

you can try to set ServerAliveInterval as "ssh arguments" in the secure-shell extension (-o ServerAliveInterval=30)

or if you set this value in your ssh config, import it like described in the secure-shell faq
https://chromium.googlesource.com/apps/libapps/+/master/nassh/doc/FAQ.md#can-i-use-my-ssh_config-file

evil-dog · Answer 2 · Wed Sep 12 2018 13:20:30 GMT+0800 (China Standard Time)

That seems to have worked. I added that parameter and it stayed alive and well for over an hour idle.

You might want to look to add this to the docs somewhere (if it's not already).

Thanks.

Lukas Prettenthaler · Answer 3 · Wed Sep 12 2018 14:34:13 GMT+0800 (China Standard Time)

i created a faq section in the wiki