Hangs

[Oliver L Schoenborn]

When I run the example (Ubuntu 18, python 3.6) it just hangs forever. Whenever I ctrl-c the process, it shows the same output:

me:~ $ python3 -m aws_list_all query --service ec2 --region us-east-1
^CTraceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 746, in next
    item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/aws_list_all/__main__.py", line 109, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/aws_list_all/__main__.py", line 87, in main
    do_query(services, args.region, args.operation)
  File "/usr/local/lib/python3.6/dist-packages/aws_list_all/query.py", line 91, in do_query
    for result in ThreadPool(32).imap_unordered(acquire_listing, to_run):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 750, in next
    self._cond.wait(timeout)
  File "/usr/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt
me:~ $ 

Direct aws CLI commands like aws s3 ls work fine.

[lmind]

+1

[Johannes Ebke]

Hmm. Unfortunately, I cannot reproduce the issue here.
"IndexError: pop from an empty deque" is just an issue with Ctrl-C'ing into the multiprocessing.

I've now added extra verbosity to the execution, and if you add "-vvv" to the run it should print out the full traceback of any exceptions. That would very much help in finding any bugs!

[Felix Huttmann]

I have the same problem using python 3.6.7, but it worked with 3.7.3, it seems a bug in python multiprocessing, workaround is here: felixhuttmann@81e45eb