A python package with container support for Toil pipelines.
Check the example! This package was built to support the cookiecutter-toil repository.
-
π¦ Easy Installation
pip install toil_container
-
π³ Container System Calls
docker_call
andsingularity_call
are functions that run containerized commands with the same calling signature. Be default theexit code
is returned, however you can get thestdout
withcheck_output=True
. You can also set theenv
,cwd
,volumes
andworking_dir
for the container call.working_dir
is used as the/tmp
directory inside the container.from toil_container import docker_call from toil_container import singularity_call cmd = ["cowsay", "hello world"] status = docker_call("docker/whalesay", cmd) output = docker_call("docker/whalesay", cmd, check_output=True) status = singularity_call("docker://docker/whalesay", cmd) output = singularity_call("docker://docker/whalesay", cmd, check_output=True)
-
π³ Container Job Class
ContainerJob
is a Toil Job Class with acall
method that executes commands with eitherDocker
,Singularity
orSubprocess
depending on image availability. Check out this simple whalesay example! The Job must be constructed with anoptions
argument of the typeargparse.Namespace
that may have the following attributes:attribute action description options.docker
use docker name or path to image options.singularity
use singularity name or path to image options.workDir
set as container /tmp
path to work directory options.volumes
volumes to be mounted list of src, dst tuples -
π Container Parser With Short Toil Options
ContainerArgumentParser
adds the--docker
,--singularity
and--volumes
arguments to the options namespace. This parser only prints the required toil arguments when using--help
. However, the full list of toil rocketry is printed with--help-toil
. If you don't need the container options but want to use--help-toil
useToilShortArgumentParser
.whalesay.py --help usage: whalesay [-h] [-v] [--help-toil] [TOIL OPTIONAL ARGS] jobStore optional arguments: -h, --help show this help message and exit --help-toil print help with full list of Toil arguments and exit container arguments: --docker name/path of the docker image available in daemon --singularity name/path of the singularity image available in deamon --volumes tuples of (local path, absolute container path) toil arguments: TOIL OPTIONAL ARGS see --help-toil for a full list of toil parameters jobStore the location of the job store for the workflow [REQUIRED]
whalesay.py
is an example that runs a toil pipeline with the famous whalesay docker container. The pipeline can now be executed with either docker, singularity or subprocess.
# whalesay.py
from toil_container import ContainerJob
from toil_container import ContainerArgumentParser
class WhaleSayJob(ContainerJob):
def run(self, fileStore):
"""Run `cowsay` with Docker, Singularity or Subprocess."""
msg = self.call(["cowsay", self.options.msg], check_output=True)
fileStore.logToMaster(msg)
def main():
parser = ContainerArgumentParser()
parser.add_argument("-m", "--msg", default="Hello from the ocean!")
options = parser.parse_args()
job = WhaleSayJob(options=options)
ContainerJob.Runner.startToil(job, options)
if __name__ == "__main__":
main()
Then run:
# run with docker
whalesay.py jobstore -m 'hello world' --docker docker/whalesay
# run with singularity
whalesay.py jobstore -m 'hello world' --singularity docker://docker/whalesay
# if cowsay is available in the environment
whalesay.py jobstore -m 'hello world'
If you want to convert a docker image into a singularity image instead of using the docker://
prefix, check docker2singularity, and use -m '/shared-fs-path /shared-fs-path'
to make sure your shared file system is mounted inside the singularity image.
Contributions are welcome, and they are greatly appreciated, check our contributing guidelines! Make sure you add your name to the contributors list:
- π Juan S. Medina @jsmedmar
- π΄ Juan E. Arango @juanesarango
- π Max F. Levine @mflevine
- πΌ Joe Zhou @zhouyangyu
- This repo was inspired by toil's implementation of a
Docker Call
and toil_vg interface ofSingularity Calls
. - This package was initiated with Cookiecutter and the audreyr/cookiecutter-pypackage project template.