Mongo sorting error after adding more samples

[Pͥetͣeͫr Zuidˢeͬmͨaͬ]

My number of samples is increasing steadily (28832) and today snake-skin stopped working due to an error from snake-core, see below. I'll have a look myself but it seems like if passed a threshold.

[E 200401 11:50:25 web:1621] Uncaught exception GET /store?limit=10&order=-1&sort=timestamp (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:5000', method='GET', uri='/store?limit=10&order=-1&sort=timestamp', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "lib/python3.7/site-packages/tornado/web.py", line 1543, in _execute
        result = yield result
      File "lib/python3.7/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "lib/python3.7/site-packages/snake-1.0.1-py3.7.egg/snake/routes/store.py", line 56, in get
        while await cursor.fetch_next:
      File "lib/python3.7/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/cursor.py", line 1073, in _refresh
        self.__send_message(q)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/cursor.py", line 955, in __send_message
        address=self.__address)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/mongo_client.py", line 1346, in _run_operation_with_response
        exhaust=exhaust)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/mongo_client.py", line 1464, in _retryable_read
        return func(session, server, sock_info, slave_ok)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/mongo_client.py", line 1340, in _cmd
        unpack_res)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/server.py", line 136, in run_operation_with_response
        _check_command_response(first)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/helpers.py", line 159, in _check_command_response
        raise OperationFailure(msg % errmsg, code, response)
    pymongo.errors.OperationFailure: Executor error during find command :: caused by :: errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."

[Pͥetͣeͫr Zuidˢeͬmͨaͬ]

Quick fix, setting 64MB for sorting.

$ mongo localhost
> use admin
> db.adminCommand({setParameter: 1, internalQueryExecMaxBlockingSortBytes:67108864})

[Alex Kornitzer]

Ooo, looks like this might up pagination to the top of the priority list, need to make use of limit and skip, which has been on the list for a while, because returning all the samples the the front end is insane. Will have a look into this when I find some time.

[Pͥetͣeͫr Zuidˢeͬmͨaͬ]

Just an update from my end, currently I have 66,360 samples and the error is back.
So I've updated MongoDB to use 78Mb for sorting.

[Alex Kornitzer]

Apologies, have been super busy and just not found time for this. Ill see if i can set some time aside this weekend.

[Pͥetͣeͫr Zuidˢeͬmͨaͬ]

I wasn’t trying to pressure you. Apologies are not needed.

[Alex Kornitzer]

Right finally found time and have a new laptop, so time to code :D

So pagination has been added to snake-core: WithSecureLabs/snake-core@14c7cf6
And I have added backend pagination support to snake-skin: WithSecureLabs/snake-skin@296420b

Not sure how you are deployed but would be great if you could test those to check that it solves the problem for you.

[Pͥetͣeͫr Zuidˢeͬmͨaͬ]

Works like a charm on my end.