Q: multiprocessing/fork

[James Addison]

Hi @ask, hope you don't mind another question. I'm hoping to fork the initial process running my Services, to take advantage of multiple cores. What is the best approach given mode's structure?

I've looked at the source of gunicorn for hints, but I'd like to get some feedback from you as well - specifically in regards to a mode compatible approach.

For context, I've got an Application class derived from mode.Worker, and a number of mode.Service subclasses as well that are initialized on Application instantiation.

I appreciate any feedback you might have, thank you.

---

Edit: I see use of concurrent.futures.ProcessPoolExecutor (https://pymotw.com/3/asyncio/executors.html#processes). For more detail, I actually have two reasons for wanting to run asyncio/mode process-based work:

• gunicorn-like handling of incoming requests so they're spread across cores
• running computationally intensive tasks in parallel across cores; I believe ProcessPoolExecutor makes sense here

Those two scenarios would be happening on separate systems (one is a frontend mode.Service providing request validation, the other is a backend mode.Service performing things like image renders)

[Ask Solem]

There's already support for running a service in a thread:
https://github.com/ask/mode/blob/master/mode/threads.py

A similar thing for running in separate processes could work (using ProcessPoolExecutor instead of the ThreadPoolExecutor).

Note though,

• multiprocessing.Pool can easily deadlock when mixed with threads
• multiprocessing.Pool uses threads in the implementation
• multiprocessing.Pool uses a POSIX semaphore to share a pipe between child processes,
  if a process is killed while having this semaphore acquired, you will get a deadlock.

The reasons above are why I spent six months rewriting multiprocessing.Pool to be async and threadless in Celery.
Sadly that code is not very reusable as is, but I think an asyncio compatible Pool class would be necessary.

You can use it as-is, just make sure to restart processes if they hang :P

[James Addison]

Thanks for the feedback. I'll see if I can figure this out!