Giters

pydoit / doit

task management & automation tool

Home Page:http://pydoit.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

support async Python actions

wahuneke opened this issue · comments

commented

Motivations

Two motivations for this (sizeable) request:

  • it may provide a simpler route to parallel action support (which is currently unsupported in Windows and has limitations elsewhere)
  • allow for integration against async libraries (e.g. aiodocker)

Usage

  1. Declaring a Python async function as a task's action would result in that task's action being pushed onto the event loop. Additional plumbing would be added to the loop as well in order to chain dependent tasks in downstream
  2. Activating a global configuration option would cause doit to automatically run all shell command blocks using an async shell exec adapter such as Python's async subprocess feature

The work

I have not looked at doit code yet, but would consider volunteering for some / all of this effort if more-experienced contributors believe it is worth pursuing.

I did some quick searches through PRs and existing issues and did not see any discussion of Python's async faculties... Sorry if I missed something or if this is already asked and answered

Fund with Polar
commented

This would definitely be a nice addition.

The code you are looking for is at runner.py.
You can see default implementation and parallel multiprocess/thread.

Actually I have done some experimentation related to this a while ago (in a different project)...
I want to move away from using the multiprocess module altogether.

I guess a first step would be to just support async actions (run just one at time).

As of today you can specify the parallel mechanism only per run (i.e. all tasks a threads or process).
Ideally would be able to group tasks and run them on different parallel strategies with master as async code.

See also

commented

Thank you for the pointers. I have looked over runner.py and can see that this may be the place to start. I also like the suggestion of starting with async actions but enforcing 'one at a time' at the task level.

I'm currently working on another doit extension... I'll return to this async idea after I get the alpha release for the other project :)