GitTeaching / concurrency_parallelism_asyncio

Multiprocessing, (multi)threading ,and asynchronous programming with asyncio. Python Event-Driven architecture using RabbitMQ and pika.

concurrency_parallelism_asyncio

Multiprocessing, (multi)threading ,and asynchronous programming with asyncio and Celery.

Python Event-Driven architecture using RabbitMQ and pika.

The basic types of concurrency available in Python:

• Multithreading
• Multiprocessing
• Asyncio

Quick Recap

• Sync: Blocking operations.
• Async: Independant and Non blocking operations.
• Concurrency: Making progress together.
• Parallelism: Making progress in parallel.

Parallelism implies Concurrency. But Concurrency doesn’t always mean Parallelism.

http://masnun.rocks/2016/10/06/async-python-the-different-forms-of-concurrency/

Python Multithreading vs. Multiprocessing V. Asyncio

if io_bound:
    if io_very_slow:
        print("Use Asyncio")
    else:
        print("Use Threads")
else:
    print("Multi Processing")

"If your code is IO bound, both multiprocessing and multithreading in Python will work for you. Multiprocessing is a easier to just drop in than threading but has a higher memory overhead. If your code is CPU bound, multiprocessing is most likely going to be the better choice—especially if the target machine has multiple cores or CPUs. For web applications, and when you need to scale the work across multiple machines, RQ library is going to be better for you." shorturl.at/ejnz1

Some readings / References:

- Threading / Multiprocessing :

Speed Up Your Python Program With Concurrency : https://realpython.com/python-concurrency/

- Asyncio :

Coroutines and tasks : https://docs.python.org/fr/3/library/asyncio-task.html

Event loop : https://docs.python.org/fr/3/library/asyncio-eventloop.html

Python Asyncio: Basic Fundamentals : https://dev.to/v_it_aly/asyncio-basic-fundamentals-4i5m

Async programming in Python with asyncio : https://dev.to/welldone2094/async-programming-in-python-with-asyncio-12dl

Speed Up Your Python Program With Concurrency : https://realpython.com/python-concurrency/

Asynchronous Python for Web Development : https://stackabuse.com/asynchronous-python-for-web-development/

Async IO en Python: une solution complète : https://www.codeflow.site/fr/article/async-io-python#_un_programme_complet_demandes_asynchrones

- Celery :

Celery Docs : https://docs.celeryproject.org/en/stable/index.html

Celery Tutorial: A Must-Learn Technology for Python Developers : https://medium.com/swlh/python-developers-celery-is-a-must-learn-technology-heres-how-to-get-started-578f5d63fab3

Setting up a task queue using Celery and RabbitMQ : https://medium.com/@krishnadey30/setting-up-a-task-queue-using-celery-and-rabbitmq-e73f8fd15de0

Celery: an overview of the architecture and how it works : https://www.vinta.com.br/blog/2017/celery-overview-archtecture-and-how-it-works/

Celery in the wild: tips and tricks to run async tasks in the real world : https://www.vinta.com.br/blog/2018/celery-wild-tips-and-tricks-run-async-tasks-real-world/

Dealing with resource-consuming (Time + Memory) tasks on Celery : https://www.vinta.com.br/blog/2018/dealing-resource-consuming-tasks-celery/

Celery tasks Checklist : https://devchecklists.com/celery-tasks-checklist/

How to Use Celery and RabbitMQ with Django : https://simpleisbetterthancomplex.com/tutorial/2017/08/20/how-to-use-celery-with-django.html

---

Python GIL - Global Interpreter Lock :

From : https://realpython.com/python-concurrency/

Concurrency Type	Switching Decision	Number of Processors
Pre-emptive multitasking (threading)	The operating system decides when to switch tasks external to Python.	1
Cooperative multitasking (asyncio)	The tasks decide when to give up control.	1
Multiprocessing (multiprocessing)	The processes all run at the same time on different processors.	Many

From https://leimao.github.io/blog/Python-Concurrency-High-Level/:

Concurrency Type	Features	Use Criteria	Metaphor
Multiprocessing	Multiple processes, high CPU utilization.	CPU-bound	We have ten kitchens, ten chefs, ten dishes to cook.
Threading	Single process, multiple threads, pre-emptive multitasking, OS decides task switching.	Fast I/O-bound	We have one kitchen, ten chefs, ten dishes to cook. The kitchen is crowded when the ten chefs are present together.
AsyncIO	Single process, single thread, cooperative multitasking, tasks cooperatively decide switching.	Slow I/O-bound	We have one kitchen, one chef, ten dishes to cook.

From : https://www.vinta.com.br/blog/2017/celery-overview-archtecture-and-how-it-works/