Upgrade from 5.5.0 causes Puma::HttpParserError: Invalid HTTP format, parsing fails with Nginx proxy

[Tyflomate]

Hello, we've tried updating puma a while ago but encountered this issue but never had the time to really investigate. Now that I have the time, it seems that it comes from the upgrade to 5.5.1. I tried upgrading to this version first and instantly, the rails server answered with this:

2023-01-17 16:46:02 +0100 HTTP parse error, malformed request ("POST /graphql" - (-)): #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>

I tried to clear my browser navigation info but still got this issue. I guess this problem comes from this: GHSA-48w2-rm65-62xx but as mentionned, I use a Nginx proxy that redirects to the puma server, as mentionned in the workarounds so I don't really know what I am doing wrong.
Here is the nginx conf running on nginx version 1.18.0 and puma 5.5.0 properly:

worker_processes  1;
error_log /dev/stdout info;

events {
    worker_connections  1024;
}


http {
    default_type  application/octet-stream;
    sendfile        on;

    ssl_session_timeout 5s;
    ssl_session_tickets off;
    ssl_session_cache off;
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    client_max_body_size 200M;
    keepalive_requests 1;
    keepalive_timeout 0;

    gzip  on;

    server {
        listen       8443 ssl;
        server_name  ***;
        ssl_certificate /opt/nginx-tls/tls.crt;
        ssl_certificate_key /opt/nginx-tls/tls.key;
        ssl_client_certificate /opt/nginx/client-ca.crt;
        ssl_verify_client on;

        location / {
            proxy_set_header "X-TLS-Serial" $ssl_client_serial;
            proxy_set_header "X-TLS-Verify" $ssl_client_verify;
            proxy_set_header "X-TLS-DN" $ssl_client_s_dn;
            proxy_set_header "X-TLS-DN-Legacy" $ssl_client_s_dn_legacy;
            proxy_set_header "X-TLS-Fingerprint" $ssl_client_fingerprint;
            proxy_set_header "X-TLS-Cert" $ssl_client_cert;
            proxy_set_header "X-TLS-Espaced-Cert" $ssl_client_escaped_cert;
            proxy_set_header "Host" $host;
            proxy_pass http://***:8080;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

I reproduced this config in my local environment with the following config:

worker_processes  1;
daemon off;
error_log /var/log/nginx/error.log;

events {
    worker_connections  1024;
}


http {
    access_log /var/log/nginx/access.log;
    default_type  application/octet-stream;
    sendfile        on;

    ssl_session_timeout 5s;
    ssl_session_tickets off;
    ssl_session_cache off;
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    client_max_body_size 200M;
    keepalive_timeout 0;

    gzip  on;

    server {
        listen 82 ssl;
        server_name 127.0.0.1;
        ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
	ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
        ssl_client_certificate /opt/nginx/client-ca.crt;
        ssl_verify_client on;

        location / {
            proxy_set_header "X-TLS-Serial" $ssl_client_serial;
            proxy_set_header "X-TLS-Verify" $ssl_client_verify;
            proxy_set_header "X-TLS-DN" $ssl_client_s_dn;
            proxy_set_header "X-TLS-DN-Legacy" $ssl_client_s_dn_legacy;
            proxy_set_header "X-TLS-Fingerprint" $ssl_client_fingerprint;
            proxy_set_header "X-TLS-Cert" $ssl_client_cert;
            proxy_set_header "X-TLS-Espaced-Cert" $ssl_client_escaped_cert;
            proxy_set_header "Host" $host;
            proxy_pass http://127.0.0.1:3000;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

But with both config I have the Parser error. I tried upgrading nginx to the latest version on my local environment but still got the issue. I also tried to upgrade puma to the latest version aswell and the error still happens too. Maybe good to mention that even with config.force_ssl = true the error still happens. Am i missing something in my nginx config ? Thanks a lot for your help !

[MSP-Greg]

@Tyflomate

I haven't had time to look at this, others may have.

What Ruby & OpenSSL version are you using? Not sure if it's important...

Typo? proxy_set_header "X-TLS-Espaced-Cert"

[Tyflomate]

Hello ! Thanks for your answer, i'm on ruby 2.7.5 and on rails 6.1.7. I created my self-signed certificate with Openssl 3.0.2.
Thanks for pointing out the type, i'll change this 👍

[Patrik Ragnarsson]

proxy_set_header "X-TLS-Cert" $ssl_client_cert;

Does it work if you remove this from your config? Maybe it contains the LF character now forbidden. The documentation says its deprecated https://nginx.org/en/docs/http/ngx_http_ssl_module.html#var_ssl_client_cert

[Tyflomate]

Oh my god yes thanks a lot ! I was going mad about this, thank you 🙏