Provide an async implementation
ggreg opened this issue · comments
The current interface returns a generator of result rows and is mostly used synchronously. Leveraging async in Python 3 would allow to better utilize resources when managing parallel and concurrent queries.
One challenge is to keep compatibility with Python 2.7.
Is current best practice to use an asyncio executor when using presto-python-client in existing asynchronous applications? Are there alternative methods?
We usually rely on a plain concurrent.futures.Executor to execute concurrent Presto queries.
Are you talking about using https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor with a concurrent.futures.Executor to do Presto queries along async code?
We'd like to move to actually use async to support higher concurrent with a lower CPU and memory footprint for some application such as shadowing queries where we run the service with hundred of processes (which has a cost in memory usage as copy-on-write when forking processes tend to still copy a lot of pages with typical Python processes). There's also context switching between threads or processes and everything else that would be better with coroutines.
Is something planned related to support async execution?