XENON1T / cax

Simple data management tool

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

"unorderable types:" error during RetryStalledTransfer

pdeperio opened this issue · comments

Periodically getting the following error:

RetryStalledTransfer: ERROR    Removed from run database.
root        : CRITICAL Exception caught from task RetryStalledTransfer
Traceback (most recent call last):
  File "/stash2/project/@xenon1t/anaconda3/envs/pax_head/lib/python3.4/site-packages/cax-4.3.14-py3.4.egg/cax/main.py", line 116, in main
    task.go(args.run)
  File "/stash2/project/@xenon1t/anaconda3/envs/pax_head/lib/python3.4/site-packages/cax-4.3.14-py3.4.egg/cax/task.py", line 61, in go
    self.each_run()
  File "/stash2/project/@xenon1t/anaconda3/envs/pax_head/lib/python3.4/site-packages/cax-4.3.14-py3.4.egg/cax/task.py", line 69, in each_run
    self.each_location(data_doc)
  File "/stash2/project/@xenon1t/anaconda3/envs/pax_head/lib/python3.4/site-packages/cax-4.3.14-py3.4.egg/cax/tasks/clear.py", line 49, in each_location
    time_made)
TypeError: unorderable types: list() > datetime.datetime()
root        : ERROR    unorderable types: list() > datetime.datetime()

while running the following command on Stash (ci-connect)

HOSTNAME=login cax --once --config /stash2/project/@xenon1t/deployHQ/cax_deploy/cax_checksum_and_clean.json

Hasn't appeared in a while.

appearing again on Midway with:

HOSTNAME=midway-login1 cax --once --run 10023 --config /project/lgrandi/xenon1t/cax/cax_AddChecksum_only.json --ncpu 1 --log-file /project/lgrandi/xenon1t/cax/10023_v6.8.0/cax.log

Error is coming from this line, and I think because some RunsDB entries appear like:

"creation_time": [
  {
    "$date": 1496196054544
  }
],

instead of

"creation_time": {
  "$date": 1496519036013
},

@coderdj I remember you mentioned some time ago that you could fix this retroactively?