question: running ProcessPoolExecutor inside web-app served by hypercorn

[Konstantin Baidin]

Hello, attempt to use ProcessPoolExecutor leads to

...
  File "/Users/baidinkn/my-service/my_service/app.py", line 133, in my_endpoint
    loop.run_in_executor(pool, long_fun)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 826, in run_in_executor
    executor.submit(func, *args), loop=self)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/process.py", line 782, in submit
    self._adjust_process_count()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/process.py", line 741, in _adjust_process_count
    self._spawn_process()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/process.py", line 759, in _spawn_process
    p.start()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \
AssertionError: daemonic processes are not allowed to have children

ThreadPoolExecutor could work here, but it doesn't suit me, because I have a CPU-bound heavy function, so it must be a separate process instead of a thread.

Example of application:

pool = ProcessPoolExecutor()

def create_app() -> Quart:
    app = Quart(__name__)
    app.config.from_object(settings.quart)
    
    @security_scheme([])
    @app.route("/my_endpoint")
    async def my_endpoint() -> ResponseReturnValue:
        loop = asyncio.get_running_loop()
        loop.run_in_executor(pool, long_fun)
        return "Done"
        
def long_fun():
    time.sleep(4)

hypercorn dbt_service/asgi:app

Is there a way to use ProcessPoolExecutor in the web-apps served by hypercorn?

[Konstantin Baidin]

A little update:
I found that this code works with Uvicorn and Daphne. I don't know what is exact difference. I suppose Uvicorn and Daphne don't spawn workers and work as a single process. That is why there is no daemonic process error.

But I would like to keep hypercorn as I use quart framework. Is there a way to run hypercorn as a single process, without spawning a worker? I don't need multiple workers.

[Umair Anis]

@pgjones Looks like Hypercorn worker processes are by default run with daemon set (ref). This prevents APIs like multiprocessing.Process / concurrent.futures.ProcessPoolExecutor from creating children processes from inside the application.

Is there any reason why worker processes need to be daemonized or if this flag could be configured? Note that this is not the case with application servers like uWSGI or uvicorn.

[Umair Anis]

@konstantin-baidin-y42 Were you able to workaround this issue?

[Konstantin Baidin]

@konstantin-baidin-y42 Were you able to workaround this issue?

Finally, I just switched my service to uvicorn, it works well for me 🤷‍♂️