ekzhang / sshx

Hey 👋 and thanks for the great project!

After several attempts to figure out how to make it work with nginx reverse-proxy, I've created half-working solution. I am providing below steps to reproduce the issue and ultimately figure out what is the problem.

`sshx-server` setup

Create and enter temp folder:

mkdir /tmp/reproduce-issue && cd /tmp/reproduce-issue

Clone and enter the sshx repository:

git clone https://github.com/ekzhang/sshx.git && cd sshx

Build frontend:

npm ci && npm run build

Build backend:

cargo build --release --bin sshx-server

Launch sshx (which will also serve frontend).:

mv target/release/sshx-server . && ./sshx-server --secret dev-secret --listen :: --override-origin https://localhost:8080

`nginx` setup

Create and enter nginx folder:

mkdir /tmp/reproduce-issue/nginx && cd /tmp/reproduce-issue/nginx

Create nginx.conf with the following content:

# Run nginx using:
#     nginx -p $PWD -e stderr -c nginx.conf

daemon off;  # run in foreground

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;

    # Locally generated SSL certificates from mkcert
    ssl_certificate ./localhost+1.pem;
    ssl_certificate_key ./localhost+1-key.pem;
    ssl_protocols	TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    # Directories nginx needs configured to start up.
    client_body_temp_path .;
    proxy_temp_path .;
    fastcgi_temp_path .;
    uwsgi_temp_path .;
    scgi_temp_path .;
    
    ###################
    client_body_timeout 5;  # <------ makes sshx client spitting error log every 5 seconds
    ###################

    # Connection upgrade variable for websocket
    map $http_upgrade $connection_upgrade {  
        default upgrade;
        ''      close;
    }

    server {
        server_name localhost 127.0.0.1;
        listen 8080 ssl http2;

        location / {
            proxy_pass   http://127.0.0.1:8051;
        }

        location /api {
            # Websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass   http://127.0.0.1:8051;
        }
    }
}

Setup local certificates with mkcert (otherwise grpc/websockets won't work) and assign correct permissions:

sudo mkcert -install && sudo mkcert localhost 127.0.0.1 && sudo chown $USER:$USER *.pem

Launch nginx:

nginx -p $PWD -e stderr -c nginx.conf

`sshx` setup

The default sshx binaries won't work, because tonic is compiled with tls-webpki-roots feature, which means that sshx will not lookup OS ca-certificates bundle, making our self-signed certificates useless and throwing the following error:

❯ ./sshx --server https://localhost:8080
2023-11-06T12:15:26.058671Z ERROR sshx: transport error

Caused by:
    0: error trying to connect: invalid peer certificate: UnknownIssuer
    1: invalid peer certificate: UnknownIssuer

In order to fix that we need to recompile sshx binary with tls-roots feature (which will lookup OS ca-certificates):

Enter sshx cloned git repository:

cd /tmp/reproduce-issue/sshx

Save the following content as tonic_fix.patch:

diff --git a/Cargo.toml b/Cargo.toml
index 4b0c58e..c028b8d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,7 +19,7 @@ rand = "0.8.5"
 serde = { version = "1.0.188", features = ["derive", "rc"] }
 tokio = { version = "1.32.0", features = ["full"] }
 tokio-stream = { version = "0.1.14", features = ["sync"] }
-tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] }
+tonic = { version = "0.10.0", features = ["tls", "tls-roots"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

and apply it like so:

git apply tonic_fix.patch

Compile modified sshx:

cargo build --release --bin sshx

Launch client:

mv target/release/sshx . && ./sshx --server https://localhost:8080

Now, here's the issue: there's no connection between server and shell and I'm unable to spawn terminal:

And once in a while sshx produces the following log:

2023-11-06T14:25:05.152762Z ERROR sshx::controller: disconnected, retrying in 1s... err=status: Internal, message: "h2 protocol error: http2 error: stream error received: unspecific protocol error detected", details: [], metadata: MetadataMap { headers: {} }

Here's the log with RUST_LOG=trace: trace_sshx.log

Any help would be much appreciated!

Adding this to the server section seems to fix it:

location /sshx.SshxService/Channel {
   grpc_pass      grpc://127.0.0.1:8051;
}

Yep, this directive fixes the issue. Thanks a lot!

Maybe we should add the working nginx.conf example to wiki?

release]$ ./sshx --server https://localhost:8003
2023-11-07T19:15:11.596704Z ERROR sshx: transport error

Caused by:
0: error trying to connect: invalid peer certificate: UnknownIssuer
1: invalid peer certificate: UnknownIssuer
[muthUn@brlb release]$

As per your suggestion i was changed the below line of code in cargo.toml file still i am getting certificate error message.

diff --git a/Cargo.toml b/Cargo.toml index 4b0c58e..c028b8d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -19,7 +19,7 @@ rand = "0.8.5" serde = { version = "1.0.188", features = ["derive", "rc"] } tokio = { version = "1.32.0", features = ["full"] } tokio-stream = { version = "0.1.14", features = ["sync"] } -tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] } +tonic = { version = "0.10.0", features = ["tls", "tls-roots"] } tracing = "0.1.37" tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

@shumvgolove

Yes, you'll have to grpc_pass all requests to suffixes of /sshx.SshxService.

Actually on the server it internally just checks if the content-type is application/grpc to decide whether to handle it as an HTTP request or a gRPC request. But since all gRPC requests go to SshxService, matching by path would work fine.

sshx/crates/sshx-server/src/listen.rs

Lines 51 to 60 in e01b75a

    
           let svc = Steer::new( 
        
               [http_service, grpc_service], 
        
               |req: &Request<Body>, _services: &[_]| { 
        
                   let headers = req.headers(); 
        
                   match headers.get(CONTENT_TYPE) { 
        
                       Some(content) if content == "application/grpc" => 1, 
        
                       _ => 0, 
        
                   } 
        
               }, 
        
           );

To clarify, you don't necessarily need to detect gRPC when you're putting a reverse proxy in front of sshx. You just need to forward HTTP/2 connections to it as the backend protocol. But Nginx doesn't let you do h2c (HTTP/2 cleartext) backend for its reverse proxy; it only has grpc_pass as a special case.

Sorry about this, Nginx is just eternally difficult to configure 😅

this is what i have in my nginx.conf . I am still facing certificate issue. would you mind sharing your nginx setup @shumvgolove

$ ./sshx --server https://localhost:8002
2023-11-09T09:39:28.982734Z ERROR sshx: transport error

Caused by:
0: error trying to connect: invalid peer certificate: UnknownIssuer
1: invalid peer certificate: UnknownIssuer

`events {}

pid nginx.pid;

http {
access_log /dev/stdout;

ssl_certificate ./localhost.pem;
ssl_certificate_key ./localhost-key.pem;
ssl_protocols	TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

client_body_temp_path .;
proxy_temp_path .;
fastcgi_temp_path .;
uwsgi_temp_path .;
scgi_temp_path .;

client_body_timeout 5;

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    server_name localhost 127.0.0.1;
    listen 8003 ssl http2;

    location /sshx.SshxService {
        grpc_pass      grpc://127.0.0.1:8051;
    }

    location /api {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $http_host;
        proxy_pass   http://127.0.0.1:8051;
    }
}

}
`

I tried without nginx . Its working fine. But i will try to see if i can able to make with nginx.

Hey 👋 and thanks for the great project!

After several attempts to figure out how to make it work with nginx reverse-proxy, I've created half-working solution. I am providing below steps to reproduce the issue and ultimately figure out what is the problem.

`sshx-server` setup

Create and enter temp folder:

mkdir /tmp/reproduce-issue && cd /tmp/reproduce-issue

Clone and enter the sshx repository:

git clone https://github.com/ekzhang/sshx.git && cd sshx

Build frontend:

npm ci && npm run build

Build backend:

cargo build --release --bin sshx-server

Launch sshx (which will also serve frontend).:

mv target/release/sshx-server . && ./sshx-server --secret dev-secret --listen :: --override-origin https://localhost:8080

`nginx` setup

Create and enter nginx folder:

mkdir /tmp/reproduce-issue/nginx && cd /tmp/reproduce-issue/nginx

Create nginx.conf with the following content:

# Run nginx using:
#     nginx -p $PWD -e stderr -c nginx.conf

daemon off;  # run in foreground

events {}

pid nginx.pid;

http {
    access_log /dev/stdout;

    # Locally generated SSL certificates from mkcert
    ssl_certificate ./localhost+1.pem;
    ssl_certificate_key ./localhost+1-key.pem;
    ssl_protocols	TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

    # Directories nginx needs configured to start up.
    client_body_temp_path .;
    proxy_temp_path .;
    fastcgi_temp_path .;
    uwsgi_temp_path .;
    scgi_temp_path .;
    
    ###################
    client_body_timeout 5;  # <------ makes sshx client spitting error log every 5 seconds
    ###################

    # Connection upgrade variable for websocket
    map $http_upgrade $connection_upgrade {  
        default upgrade;
        ''      close;
    }

    server {
        server_name localhost 127.0.0.1;
        listen 8080 ssl http2;

        location / {
            proxy_pass   http://127.0.0.1:8051;
        }

        location /api {
            # Websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $http_host;
            proxy_pass   http://127.0.0.1:8051;
        }
    }
}

Setup local certificates with mkcert (otherwise grpc/websockets won't work) and assign correct permissions:

sudo mkcert -install && sudo mkcert localhost 127.0.0.1 && sudo chown $USER:$USER *.pem

Launch nginx:

nginx -p $PWD -e stderr -c nginx.conf

`sshx` setup

The default sshx binaries won't work, because tonic is compiled with tls-webpki-roots feature, which means that sshx will not lookup OS ca-certificates bundle, making our self-signed certificates useless and throwing the following error:

❯ ./sshx --server https://localhost:8080
2023-11-06T12:15:26.058671Z ERROR sshx: transport error

Caused by:
    0: error trying to connect: invalid peer certificate: UnknownIssuer
    1: invalid peer certificate: UnknownIssuer

In order to fix that we need to recompile sshx binary with tls-roots feature (which will lookup OS ca-certificates):

Enter sshx cloned git repository:

cd /tmp/reproduce-issue/sshx

Save the following content as tonic_fix.patch:

diff --git a/Cargo.toml b/Cargo.toml
index 4b0c58e..c028b8d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,7 +19,7 @@ rand = "0.8.5"
 serde = { version = "1.0.188", features = ["derive", "rc"] }
 tokio = { version = "1.32.0", features = ["full"] }
 tokio-stream = { version = "0.1.14", features = ["sync"] }
-tonic = { version = "0.10.0", features = ["tls", "tls-webpki-roots"] }
+tonic = { version = "0.10.0", features = ["tls", "tls-roots"] }
 tracing = "0.1.37"
 tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }

and apply it like so:

git apply tonic_fix.patch

Compile modified sshx:

cargo build --release --bin sshx

Launch client:

mv target/release/sshx . && ./sshx --server https://localhost:8080

Now, here's the issue: there's no connection between server and shell and I'm unable to spawn terminal:

And once in a while sshx produces the following log:

2023-11-06T14:25:05.152762Z ERROR sshx::controller: disconnected, retrying in 1s... err=status: Internal, message: "h2 protocol error: http2 error: stream error received: unspecific protocol error detected", details: [], metadata: MetadataMap { headers: {} }

Here's the log with RUST_LOG=trace: trace_sshx.log

Any help would be much appreciated!

Thanks for your guide, my problem is solved.

By the way, self signed cert need to add into keychain in Mac OS

	let svc = Steer::new(
	[http_service, grpc_service],
	\|req: &Request<Body>, _services: &[_]\| {
	let headers = req.headers();
	match headers.get(CONTENT_TYPE) {
	Some(content) if content == "application/grpc" => 1,
	_ => 0,
	}
	},
	);

nginx: stream error received: unspecific protocol error detected

sshx-server setup

nginx setup

sshx setup

sshx-server setup

nginx setup

sshx setup

`sshx-server` setup

`nginx` setup

`sshx` setup

`sshx-server` setup

`nginx` setup

`sshx` setup