newer tomcat rejects [:]

Question

newer tomcat rejects [:]

gajowi opened this issue 6 years ago · comments

Historically, thredds supports opendap subsetting urls including syntax with [:]. Recent versions of tomcat reject this by default.

See: https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
https://stackoverflow.com/questions/41053653/tomcat-8-is-not-able-to-handle-get-request-with-in-query-parameters
https://stackoverflow.com/questions/50361171/how-to-allow-character-in-urls-for-tomcat-8-5

For unidata I think there are two actions:

update server.xml in thredds-docker (to use/set at least relaxedQueryChars='[]:')
update general thredds deployment instructions

Other characters may also be in scope. I just hit the problem with opendap subsetting ascii response with 5.0beta5

Julien Chastang · Answer 1 · Mon Nov 19 2018 23:55:29 GMT+0800 (China Standard Time)

Thanks for the heads up. I'll look at this shortly. (It's Thanksgiving week here in the US).

Julien Chastang · Answer 2 · Wed Nov 28 2018 06:00:52 GMT+0800 (China Standard Time)

I see how this can give you problems, but is there anything specific to docker here? If you would like to adjust server.xml with any customizations, you can do that by mounting over the server.xml. I do just that here. BTW, this project does not contain a server.xml file. That comes from the parent tomcat-docker project which is meant to be generic, and not specific to the TDS.

It seems like this could be a point of clarification in the TDS documentation, if it is not already. Ping @lesserwhirls.

Sean Arms · Answer 3 · Wed Nov 28 2018 07:11:59 GMT+0800 (China Standard Time)

We discussed this issue over on the THREDDS issue tracker at Unidata/thredds#1144, although it's a bit lengthy and has some red herrings on our part.

Perhaps most relevant is this particular comment: Unidata/thredds#1144 (comment)

We decided that unless there is an explicit need for a server to allow unencoded urls, we suggest not using the relaxedQueryChars (which is why we haven't documented it as part of the setup/tutorial). We will point out the relaxedQueryChars option in our release announcement, however.

There were some issues with the opendap html page / javascript not encoding the url properly before submitting a request, and this has been fixed in beta6 of 5.0 and the soon-to-be-released 4.6.12.

Interestingly enough, when using an Apache server as a front-end to Tomcat, the illegal characters are handled without issue. This is the setup we use at Unidata, and so we had no idea there was an issue...I expect Apache will start disallowing these too at some point soon.

Julien Chastang · Answer 4 · Thu Mar 14 2019 05:39:04 GMT+0800 (China Standard Time)

Unless, there are objections, I am going to close this issue as it relates more to the core TDS project than this docker container. In addition, it was discussed more recently in the 4.6.13 release notes.

Julien Chastang · Answer 5 · Thu Mar 21 2019 02:04:34 GMT+0800 (China Standard Time)

Ugh. I just ran into this issue myself and as this docker container does not use Apache as the front end, I think, at the very least, it needs to be better documented and any example server.xml needs to have a relaxedQueryChars as part of it.

Kyle Wilcox · Answer 6 · Tue Sep 10 2019 02:37:40 GMT+0800 (China Standard Time)

@julienchastang any chance you re-visit this and make a "working" server.xml part of this project? I maintain an ERDDAP docker container and recently did the same. The other option is that everyone maintains a custom server.xml files instead of the batteries being included here.

axiom-data-science/docker-erddap@32ba572#diff-4acbaad6af43f602bbb598fba00d43e0R72-R73

Tom Cook · Answer 7 · Tue Sep 10 2019 03:10:29 GMT+0800 (China Standard Time)

Thanks for mentioning this Kyle. I agree it would be much easier since I don't like the idea of editing the server.xml inside the container. I thinking having users maintain their own server.xml might get messy, and really other than the relaxed characters setting, I never touch the server.xml

Shane St Savage · Answer 8 · Tue Sep 10 2019 03:37:13 GMT+0800 (China Standard Time)

Running this inside the container (e.g. in $CATALINA_HOME/bin/setenv.sh, if you're already mounting a custom setenv.sh) will update server.xml to accept path characters []| and query characters []|{}^\`"<> without having to mount a custom server.xml.

if ! grep relaxedPathChars $CATALINA_HOME/conf/server.xml > /dev/null; then
  echo "Updating server.xml to allow for special URL characters (needed for DAP requests)"
  echo "See https://github.com/Unidata/thredds-docker/issues/209"
  echo "and https://github.com/Unidata/thredds/releases/tag/v4.6.13"
  sed -i 's~\(port="8080" protocol="HTTP/1.1"\)$~\1 relaxedPathChars="[]|" relaxedQueryChars="[]|{}^\&#x5c;\&#x60;\&quot;\&lt;\&gt;"~' \
    $CATALINA_HOME/conf/server.xml
fi

Julien Chastang · Answer 9 · Wed Sep 11 2019 00:36:07 GMT+0800 (China Standard Time)

&#x5c is \

&#x60 is `

&quot is "

&lt is <

&gt is >

We will still need these, right?

[ [

] ]

: :

Julien Chastang · Answer 10 · Wed Sep 11 2019 01:07:37 GMT+0800 (China Standard Time)

Is something like this what you have in mind Unidata/tomcat-docker#69?

Slight variation of axiom-data-science/docker-erddap@32ba572#diff-4acbaad6af43f602bbb598fba00d43e0R72-R73

with the :

Julien Chastang · Answer 11 · Thu Sep 12 2019 00:58:41 GMT+0800 (China Standard Time)

Closing with Unidata/tomcat-docker#69. Let me know if there are any additional issues.