Mixed content errors with reverse proxy and HTTPS
tari opened this issue · comments
Running Airsonic 10.0.1 on Linux behind an Apache reverse proxy; some views attempt to load over http and are blocked as mixed content, whereas others (like About or artist/album views) are not.
For instance, clicking Settings I get a warning in my browser console (domains changed for public consumption):
Blocked loading mixed active content http://example.net/subsonic/musicFolderSettings
Inspecting the request as logged by the browser, the server is indeed returning a Location with HTTP:
GET https://example.net/subsonic/settings.view? [HTTP/1.1 302 26ms]
Location http://example.net/subsonic/musicFolderSettings
Solutions like #435 don't seem to help; my server configuration looks like this (some irrelevant parts removed):
Listen 443
<VirtualHost *:443>
SSLEngine on
SSLCertificateFile "/etc/letsencrypt/live/..."
SSLCertificateKeyFile "/etc/letsencrypt/live/..."
ProxyRequests off
ProxyPreserveHost on
SSLProxyEngine on
Header always set Strict-Transport-Security "max-age=604800"
RequestHeader set X-Forwarded-Proto "https"
ProxyPass /subsonic/ http://localhost:4040/subsonic/
ProxyPassReverse /subsonic/ http://localhost:4040/subsonic/
</VirtualHost>
And server.contextPath is set to /subsonic.
Could quickly test if your context path is correctly set ?
https://airsonic.github.io/docs/configure/standalone/#context-path
I notice the docs say to set server.context-path and I had server.contextPath, but they appear to behave the same. Checking the command line of my running server, it is set as expected:
java -Xmx512m -Dserver.address=127.0.0.1 -Dserver.port=4040 -Dserver.httpsPort=0 -Dserver.context-path=/subsonic ...
No change in behavior if I set context-path or contextPath (I guess Spring treats them equivalently?).
To debug proxy issues you can increase the logging on org.airsonic.player.service.NetworkService. One way to do this is using an additional java system property -Dlogging.level.org.airsonic.player.service.NetworkService=trace.
Assuming you're a developer type, you can also just set a debugger in the NetworkService and see what happens in there.
Nothing useful logged by the NetworkService, I'm only seeing messages from it in response to requests to dwr/call/plaincall/nowPlayingService.getNowPlayingForCurrentPlayer.dwr, and those have the correct base URL. It prints nothing if I try to open settings, for instance.
I have noticed that opening settings via the "Settings" link in the web player at the bottom of the page (playerSettings.view) opens settings correctly, whereas the Settings button at the top (settings.view) doesn't work. It looks like playerSettings isn't redirecting, whereas settings is (and generating the wrong target).
SettingsView returns a RedirectView to either musicFolderSettings or userSettings; the Spring documentation notes that RedirectView is not context-relative by default:
Note that while the default value for the "contextRelative" flag is off, you will probably want to almost always set it to true. With the flag off, URLs starting with "/" are considered relative to the web server root, while with the flag on, they are considered relative to the web application root. Since most web applications will never know or care what their context path actually is, they are much better off setting this flag to true, and submitting paths which are to be considered relative to the web application root.
I suspect using the two-parameter constructor to set contextRelative may fix the problem.
Changing to new RedirectView(..., true) doesn't fix it, but the problem does seem to only happen where redirects are involved.
A little late to the party, but I just realized this is probably the same issue I've had since my old 5.3 server.
@tari have you attempted configuring your server for HTTPS according to this page?
http://www.subsonic.org/pages/getting-started.jsp#4
The subsonic documentation isn't helpful because it describes making the server do HTTPS, whereas I want my proxy to handle that.
However, I think I found the magic switch: setting server.use-forward-headers=true appears to make redirects work correctly.
I tried adding this line to my service file under the ExecStart, did 'systemctl daemon-reload' and then restarted the service ('service airsonic restart'), but it still shows the same error. Did I miss some step, perhaps?
What does your proxy config look like? You may be missing some options that are important to making it all work.
<IfModule mod_ssl.c>
<VirtualHost *:443>
ProxyPreserveHost On
ProxyRequests Off
ServerName domain
ServerAlias domain
ServerAdmin webmaster@localhost
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://127.0.0.1:8080/
#RewriteEngine on
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
SSLCertificateFile /etc/letsencrypt/live/domain/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>```
You probably need to set X-Forwarded-Proto (see my config at the top); the use-forward-headers option makes Spring honor X-Forwarded-Proto, but Apache doesn't set it automatically.
Yes! That did it. Thank you 😃
I was encountering the same mixed content errors. Firefox seems to be more strict than Chrome here. The solution was to set ProxyPassReverse to the external address like so:
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://airsonic.mydomain.com/
RequestHeader set X-Forwarded-Proto "https"
Setting the RequestHeader alone didn't work for me. See also: https://stackoverflow.com/questions/24351782/proxypassreverse-dropping-https#27155124
Use of ProxyPreserveHost seems to be a confounding factor. If it's On, then changing ProxyPassReverse to refer to the external domain name works because Apache will pass the external host down to Airsonic which will in turn serve responses with the external hostname.
If you disable ProxyPreserveHost, setting ProxyPreserveHost to match ProxyPass will work because the request made to Airsonic has a Host: header containing the internal hostname which will be repeated in responses and correctly rewritten.
I think where this confuses a lot of people (causing the general problem) is that the docs say Airsonic expects X-Forwarded-* headers to be set, but the effect of setting X-Forwarded-Proto is not evident because the only place it's used is in NetworkService.getBaseUrl. This means the server generates headers that ignore X-Forwarded-Proto but rendered pages (and more generally, URLs generated by Airsonic itself rather than Spring) have the correct external URL.
This behavior seems inconsistent and confusing to me. The server should either always generate URLs consistent with the fields reported by the proxy or never, and it's not reasonable to never do it because proxy servers in general won't rewrite HTML.
@tari, I agree that is not the desired behavior and is inconsistent. Can you describe your setup in detail a bit more? Just want to see what you're running into. Are you running proxy and airsonic on separate hosts? Which distro?
They're both on the same host, and I'm on Arch. Switching ProxyPreserveHost off in my Apache configuration does make it work correctly, as does changing ProxyPassReverse to refer to the external hostname.
This doesn't work unless I turn on server.use-forward-headers:
ProxyPreserveHost on
ProxyPassReverse /subsonic/ http://localhost:4040/subsonic/
These both work without server.use-forward-headers:
ProxyPreserveHost off
ProxyPassReverse /subsonic/ http://localhost:4040/subsonic/
ProxyPreserveHost on
ProxyPassReverse /subsonic/ http://example.net/subsonic/
I'm always reaching the server as https://example.net/subsonic/.
@muff1nman let me stress out that if you're accessing it via https://localhost domain, the browser policy triggering this issue is not activated. It must be some other name, preferably a real domain with a valid TLS certificate set up in proxy
Okay I think I've gotten to the bottom of this. When testing with https terminated by httpd on one machine which proxies to another on http, I got the blocked content error with the latest release. However, with master I do not get the blocked issue. The reason is because of 3c95553 which had caused 10.1.0 to be released with jetty instead of tomcat. Jetty does seem to need the useForwardHeaders, however Tomcat's behavior is not changed by that setting as per Springs internal usages of org.springframework.boot.autoconfigure.web.ServerProperties#getOrDeduceUseForwardHeaders.
However, just testing with tomcat I did find an interesting behavior where the login page redirection incorrectly redirects one to a http login page even though your initial url was https. This would probably be masked by those having a redirect from http to https on their proxy. I created #655 to fix that problem.
So I think this can be brought to a close as it should be fixed in the next release. I think I will leave #655 out of the next patch fix to get more testing/feedback on it.
Okay looks like I'm not a very good tester :/ In the above, I tested using an external tomcat, not the embedded one. And looking at it further it it looks like the Tomcat RemoteIPValve does a bit more than just set the remote ip - it also sets the scheme. Looking into this a bit further.
Actually it looks like ProxyPreserveHost on messes with the ProxyPassReverse option. Turning that off made things work. The httpd docs have this to say about ProxyPreserveHost:
When enabled, this option will pass the Host: line from the incoming request to the proxied host, instead of the hostname specified in the ProxyPass module="mod_proxy" line.
This option should normally be turned Off. It is mostly useful in special configurations like proxied mass name-based virtual hosting, where the original Host header needs to be evaluated by the backend server.
We should have no logic looking at the Host header.
So with removing that option and a new release this issue should be resolved - I think :)
I'm actually getting 403 errors when searching now:
"Invalid CSRF Token '567f994e-5077-49c6-a325-b86e2bbd13a1' was found on the request parameter '_csrf' or header 'X-CSRF-TOKEN'."
So hopefully it will be resolved in the new version :)
For anyone looking here for a solution:
If you are using systemd, add server.use-forward-headers=true to airsonic.service:
ExecStart=/usr/bin/java \
$JAVA_OPTS \
-Dairsonic.home=${AIRSONIC_HOME} \
-Dserver.context-path=${CONTEXT_PATH} \
-Dserver.port=${PORT} \
-Dserver.use-forward-headers=true \
-jar ${JAVA_JAR} $JAVA_ARGS
This got resolved for me somewhere in Feb/March (using the linuxserver/airsonic docker container).
I've tried every single combination I've run across, but I cannot get this to work using the linuxserver/airsonic container. Mixed content warnings every time.
I've tried adding -Dserver.use-forward-headers=true to my command line and my proxy (Traefik) is passing these headers:
X-Forwarded-Host: airsonic.my.tld
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik
X-Real-Ip: 75.xxx.195.103
The only thing I haven't been able to try is getting the redirect to return a 301—it gives me a 302 every time and none of the documented settings will change it.
Can anyone offer any advice?
Okay, I checked my container just now and this is broken yet again:
I use Clementine/Audinaut on desktop/mobile so didn't notice when it broke.
@captn3m0 I upgraded to 1.6.0-rc6 of Traefik but still can't get it to send a 301 even with traefik.frontend.redirect.permanent=true.
I don't really think it'll fix the problem but it's literally the only thing I haven't tried at this point. I mostly want to use the server for the API like you are, but it really annoys me that I can't get this to work.
server.use-forward-headers looks at X-Forwarded-Proto and X-Forwarded-For, where you don't say the latter is being set so it seems reasonable that use-forward-headers wouldn't do anything for you.
On a quick glance (not being familiar with Traefik), I don't see a way to control those headers but maybe you can.
@RulerOf the 301 trick won't work because it gets picked up by the browser as a CORS request, and it sends a OPTIONS, which doesn't work with 301
Airsonic just needs to use relative URLs, that is the cleanest way of fixing this
@tari I must've missed a line on the copy/paste or something—I checked again and I do see an X-Forwarded-For header reaching my containers as well in the raw output:
X-Forwarded-For: 75.118.21.81
X-Forwarded-Host: whoami.rulerof.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik
X-Real-Ip: 75.118.21.81
My airsonic command line looks like this:
[root@drew-metal content-engine]# ps aux | grep airsonic
<snip>
airsonic 13454 1.3 3.5 28496080 3492192 ? Ssl 03:02 5:05 java -Dorg.eclipse.jetty.annotations.maxWait=1800 -Dairsonic.home=/config -Dairsonic.defaultMusicFolder=/music -Dairsonic.defaultPodcastFolder=/podcasts -Dairsonic.defaultPlaylistFolder=/playlists -Dserver.use-forward-headers=true -jar /usr/share/java/jetty-runner.jar --host 0.0.0.0 --port 4040 --path / airsonic.war
Are there any other headers I should set?
@captn3m0 that's unfortunate, but at least I can stop trying to make the 301 redirect happen since it won't work! 😄
@RulerOf I think you should try to run airsonic using the embed Tomcat server instead of using the Jetty server.
Also I suggest to open a second ticket to have a better track of the different issues.
@jooola I think that's probably a good troubleshooting step as well. Once I figure a few more things out, I'll open up a fresh issue wherever it should go. This may just be a problem with the linuxserver/airsonic implementation of the app.
FWIW, a little while back I managed to fix the mixed content warnings using the linuxserver/airsonic container combined with the linuxserver/letsencrypt container. Below are some of the config settings; I don't totally know what's relevant here, to be honest.
For the airsonic container, I use: CONTEXT_PATH: '/'
Then in the various letsencrypt conf files that get used:
proxy.conf:
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect http:// $scheme://;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_cache_bypass $cookie_session;
proxy_no_cache $cookie_session;
proxy_buffers 32 4k;
airsonic.conf:
server_name airsonic.my.tld;
proxy_pass http://192.168.2.168:4040;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@captn3m0 @jooola Reporting back
I finally got this to work today after switching to the first-party airsonic/airsonic container, since it uses the built-in tomcat that you suggested, @jooola.
My Airsonic server does not exhibit the mixed content warning anymore.
I'm still using - JAVA_OPTS=-Dserver.use-forward-headers=true -Dserver.context-path=/ in my docker-compose.yml. I'm not certain it's necessary to specify them, but I'm kinda "ain't broke, don't fix" about it for now. I'm willing to investigate their necessity if needed, just LMK.
@RulerOf unsure which of the two JAVA_OPTS does it, or maybe both, but can confirm they're working, adding - JAVA_OPTS=-Dserver.use-forward-headers=true -Dserver.context-path=/ finally solved the issue for me.
But only for the airsonic/airsonic docker container, not the linuxserver/airsonic!
I'm experiencing this problem on v10.1.1 with the standalone war. Am I to understand from the conversation in this thread that Airsonic will only function correctly behind a proxy that rewrites response headers (ie - Apache with ProxyPassReverse)?
I managed to get this working with traefik and the linuxserver/airsonic docker by adding the server.use-forward-headers=true to the airsonic.properties file inside /config folder. Stop the container, edit the file, start the container.
I managed to get this working with traefik and the linuxserver/airsonic docker by adding the server.use-forward-headers=true to the airsonic.properties file inside /config folder. Stop the container, edit the file, start the container.
This finally solved my problem. Thank you @imro2
Actually I use a NginX/Docker setup, but the same steps was needed in my case, and I tried to create a small pull request to the NginX configuration: Update nginx.md #52
I did read the guide lines but I still ask you to excuse most of my errors, as it is my first pull request :)
I have a similar problem...
Server:
OS: Ubuntu 18.04 LTS Server (Up-to-date)
Airsonic Version: Docker Image downloaded 18.04.2019
Webserver & Proxy: Apache
What I have done:
- Added the following lines to my Apache SSL Config:
SSLProxyEngine on
ProxyPass /airsonic http://servername.wtf:1234/airsonic
ProxyPassReverse /airsonic http://servername.wtf:1234/airsonic
RequestHeader set X-Forwarded-Proto "https"
-
Complete (Cleaned) Apache Config: ssl.conf_cleaned.txt
-
Added
server.use-forward-headers=truetoairsonic.propertiesinside the docker container.
What I expected:
When I navigate to https://servername.wtf/airsonic I get forwarded to the Airsonic Site.
What happened:
Firefox shows the following error Message: SSL_ERROR_RX_RECORD_TOO_LONG
As far as I know, this error is about sending http data through https.
Any further Idea how to solve this Problem?
At a guess the problem is enabling SSLProxyEngine which is telling Apache to use HTTPS to connect to the backend, but then you tell it to connect over plain HTTP. But this seems unrelated to the parent issue.
But that is exactly what it says in the manual: https://airsonic.github.io/docs/proxy/apache/
So it does, and I see my configuration contains that as well so it's probably not the problem. You'll probably be better off opening a new issue asking for help on this, though I suspect it's an Apache configuration problem rather than anything Airsonic-specific.
I fixed this on my Apache proxy by setting upgrade-insecure-requests. e.g. Header always set Content-Security-Policy: upgrade-insecure-requests