Rich Stoner, 2014
A simple execution environment that sits on top of girder + ipython + celery + magic.
What is this?
This is an arbitrary task with hooks into girder, that you can write in ipython, that can be run asynchronously via celery and managed in flower. After installing the items in this repo to their respective locations, you should be able to run any ipython notebook cell as a celery task by simply adding %%girder to the beginning of the cell.
This uses a custom ipython magic command to handle the imports and redirect the cell contents to a running celery worker. Unfortunately, we're only able to return the task id.
-
Prototype code in ipython, directly accessing girder models.
-
With a bit of magic, run the same code on a celery queue
-
Add hooks and run the scripts from the interface (ala romanesco without the typing)
Thanks to some intelligent path settings, we can import the girder module directly, enabling us to access models with ease.
We can also use the prebuilt girder client as a way to abstract out the commands.
-
Custom ipython notebook configuration with appropriate imports
-
Custom ipython notebook magic, to re-route commands to celery
-
A generalized execute celery execute task
-
A lightweight girder plugin to call the task from post data
Currently, proletariat has heavy dependencies on my application hosting framework (Simple application Framework) and Kitware's girder. For more information, please visit:
Side note: Kitware has put together something similar that integrates with Girder. It's called Romanesco (http://romanesco.readthedocs.org/en/latest/) and operates similar to my approach, but uses a bit strong typing and integration with girder. It is an integral part of their Tangelo platform: http://tangelo.kitware.com/ & https://github.com/tangelo-hub.