Error during sync
universalinterchange opened this issue · comments
Get the following error after it sync's for a while.
Unexpected error occurred, stopping!
Traceback (most recent call last):
File "/home/mark/.local/bin/drpg", line 8, in
sys.exit(run())
^^^^^
File "/home/mark/.local/lib/python3.11/site-packages/drpg/cmd.py", line 32, in run
DrpgSync(config).sync()
File "/home/mark/.local/lib/python3.11/site-packages/drpg/sync.py", line 67, in sync
pool.starmap(self._process_item, process_item_args)
File "/usr/lib64/python3.11/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
File "/usr/lib64/python3.11/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mark/.local/lib/python3.11/site-packages/drpg/sync.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mark/.local/lib/python3.11/site-packages/drpg/sync.py", line 76, in _process_item
url_data = self._api.file_task(product["products_id"], item["bundle_id"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mark/.local/lib/python3.11/site-packages/drpg/api.py", line 94, in file_task
while (data := resp.json()["message"])["progress"].startswith("Preparing"):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
TypeError: string indices must be integers, not 'str'
That looks like an unhandled 400 response from DriveThruRPG API.
Thanks for reporting this! I'll try to reproduce and fix this.
It works for me 😕
I can prepare a branch for you with extra logging so we can understand what precisely is happening.
You can also try running it with --log-level=DEBUG and pasting the output here.
Not sure if this makes a difference or not, but I have purchased a considerable amount of products from DTRPG. Something on the order of 50K+ files. It always dies after 33K files. I'm running this on Fedora 37 with python 3.11.1
How much of the debug log do you want? It's about 2.6M
@universalinterchange I need to prepare a new release for you to catch this exception and log more info, hope to do it this week.
I threw in a little debug code and resp.json["message"] when it errors is:
Connection refused by endpoint: 404 - {"error":"The requested product does not appear in your order history."}
But I don't know how to skip the product on that error
Thank you for digging into it. I'll try to prepare a new release based on what you've discovered.
Do you know what product is causing it - or rather why it is happening on this product? For example, if you go to My Library on your DriveThruRPG account, is it visible there?
It appears to be a Product that was a Bundle from the WargamersVault that isn't visible on the DTRPG site. All of the bundle subproducts are still available. https://www.wargamevault.com/product/392924/OBSCURE-BATTLES-3--DRONE-COMMANDER-SQUADRON-BUNDLE. I wonder if it's a database integrity issue where the bundle was deleted from one table but still represented in another.
@universalinterchange Would you mind installing code from #35 (commit b0163b5) ie by running: pip install https://codeload.github.com/glujan/drpg/zip/b0163b5085007993ce1728a3aaa7809d4b6c9a2f and trying out if this patch fixes the issue?
That worked... thank you