Searches are now limited to cookie length; error: ActionDispatch::Cookies::CookieOverflow
makyen opened this issue · comments
Here is an example search which results in this error <--- [That link reproduces the error for me in my main profile, but not in my Maktest test profile this link reproduces the error in all profiles I've tried it in. The original link I used here was just causing the error in my primary profile. See below for a bit more detail.]
In the past, I've been able to run some rather large regexes (a few examples of some moderate to long regexes, several thousand characters in some instances, from 2018) through MS search, which MS has handled well (i.e. actually performed the search and returned results).1
Today I encountered what appears to be a limitation on the overall length of all parameters passed to the regex search. It appears to be that the search parameters are being serialized, encrypted, and signed and compared to the MAX_COOKIE_SIZE and, multiple times, got a ActionDispatch::Cookies::CookieOverflow in SearchController#index error with the following code shown:
Extracted source (around line #649):
options[:value] = @encryptor.encrypt_and_sign(serialize(options[:value]), expiry_options(options)) raise CookieOverflow if options[:value].bytesize > MAX_COOKIE_SIZE end def parse_legacy_signed_message(name, legacy_signed_message)
When writing a basic search with a single title regex criteria, the new maximum length of that title regex is 1965 characters, as 1966 characters generates the error. It's actually worse than this might make it appear, because the limitation isn't on each parameter, but rather across all parameters (values and name of the parameters). For watches, keyword blacklists, and blacklisted websites, the regex is repeated 3 times (title, body, and username), so the limit for checking one of those is about 1/3 of the number above.
I'm assuming that this limitation was introduced when we updated dependencies, but I'm actually not sure when it was introduced.
As to real world usage: I encountered this error when checking on the search being done for a spam wave. In order to use a search for the regex which is currently in use as a spam wave, it was necessary to split the main regex into two separate regexes and run two separate searches.
Trace:
actionpack (5.2.6) lib/action_dispatch/middleware/cookies.rb:649:in `commit' actionpack (5.2.6) lib/action_dispatch/middleware/cookies.rb:484:in `[]=' actionpack (5.2.6) lib/action_dispatch/middleware/session/cookie_store.rb:116:in `set_cookie' rack (2.2.3) lib/rack/session/abstract/id.rb:403:in `commit_session' rack (2.2.3) lib/rack/session/abstract/id.rb:268:in `context' rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call' actionpack (5.2.6) lib/action_dispatch/middleware/cookies.rb:670:in `call' actionpack (5.2.6) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' activesupport (5.2.6) lib/active_support/callbacks.rb:98:in `run_callbacks' actionpack (5.2.6) lib/action_dispatch/middleware/callbacks.rb:26:in `call' actionpack (5.2.6) lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call' actionpack (5.2.6) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call' railties (5.2.6) lib/rails/rack/logger.rb:38:in `call_app' railties (5.2.6) lib/rails/rack/logger.rb:26:in `block in call' activesupport (5.2.6) lib/active_support/tagged_logging.rb:71:in `block in tagged' activesupport (5.2.6) lib/active_support/tagged_logging.rb:28:in `tagged' activesupport (5.2.6) lib/active_support/tagged_logging.rb:71:in `tagged' railties (5.2.6) lib/rails/rack/logger.rb:26:in `call' actionpack (5.2.6) lib/action_dispatch/middleware/remote_ip.rb:81:in `call' actionpack (5.2.6) lib/action_dispatch/middleware/request_id.rb:27:in `call' rack (2.2.3) lib/rack/method_override.rb:24:in `call' rack (2.2.3) lib/rack/runtime.rb:22:in `call' activesupport (5.2.6) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call' actionpack (5.2.6) lib/action_dispatch/middleware/executor.rb:14:in `call' actionpack (5.2.6) lib/action_dispatch/middleware/static.rb:127:in `call' rack (2.2.3) lib/rack/sendfile.rb:110:in `call' rack-cors (0.4.1) lib/rack/cors.rb:81:in `call' rack-mini-profiler (1.1.6) lib/mini_profiler/profiler.rb:296:in `call' railties (5.2.6) lib/rails/engine.rb:524:in `call' /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads' /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
- There have been times when the search has taken substantial resources, taken a long time, or even caused problems as the result of bad regex design (e.g. catastrophic backtracking) and resource loading.
A quick search indicates "Cookie overflow in rails application?" might provide a solution.
I mean... looks like that answer might work as a solution, but the real question is why Rails is storing GET parameters in a cookie at all
@ArtOfCode- Strangely, I get the error with my original error link using my primary Firefox profile, but not when I use a different profile that uses my Maktest sock account. Here is a link which reproduces the error on all profiles I've tried, and which has an extra couple thousand characters even above what was necessary, just to, hopefully, be certain of reproducing the issue for everyone.
It looks like the issue with the example URL not triggering the error is that my primary profile has a __profilin cookie, that my Maktest profile doesn't have, which is taking up 37 bytes. I'd copied the example from the URL I'd determined just barely triggers the error, so any user without that small extra __profilin cookie wouldn't see the error. I should have used a URL which was way over the limit.
I've also edited the original comment here to include a note with a URL which will actually result in the error.
@ArtOfCode- As to why Rails is storing GET parameters in the cookie, or at least requiring them to fit into that length, I have no idea. My first response when I saw that it was doing so was to wonder why it was doing so, as there doesn't seem any reason to store those in the cookie, or to check their length against possibly storing them in the cookie.
Apparently some code is storing the entire path into user_return_to:
Session dump:
_csrf_token: "lkQte4+O83YqqPqdTbTVWHt+vcHTVnE7bmLDhbPOgtY="
session_id: "5936edfd4e76a21a10eb178ff3aa1bed"
user_return_to: "/search?utf8=%E2%9C%93&title=1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa54q3gh645egh6j754hg4h5j6kfrthytgtjhegtytkjrhejk7654gtjk76hgrtyjklik76hgevebtjkl9%3B78kjrhevbnm%2C.op%3Bolkyutrvsdvbgnmytnjbhvubnmkljbvtcrdfghinmvcytrxdf76ghjiomiobygvdrs56f7ghujonbytyrd6f6ghjoimnbvtyrdf7ghuyutf676ghujouygft567g8ho"
If it is our code doing this, maybe we can check the length of url before storing it into session? If it is some library, hopefully we can monkey patch it...
Edit:
A quick grep -rnis return_to gives no hit in our code, so chances are some library (I suspect devise) is doing this.
Edit 2:
Actually, it is our code doing that...