Support Azure Batch as additonal executor
tbugfinder opened this issue · comments
New feature - Azure Batch Support
Microsoft Azure is still a growing compute environment. Azure Batch itself does support singularity and docker images and containers. Especially singularity is a unique feature.
Usage scenario
Execute container workload within Azure.
Suggest implementation
add additional executor by using the azure-java-sdk.
@tbugfinder this would be a useful feature. Many institutes rely on Azure (as an extension to all others office 365 services).
On the other way, Azure also offers a free Kubernete service (you pay for the nodes). Did you already evaluate what would be the advantages of using Azure batch over a more standard Azure Kubernetes service?
Any thoughts on this? from Nextflow team? I agree with the utility of this, my institute requires Azure for any cloud resources!
Not in the very short term but there are some chances to have in the second half of the year.
@pditommaso just wanted to check in to see if there is any progress here. Thank you!
Still pending :/
@pditommaso to get this to work we would have to use Azure BATCH API
and mimic AWS Batch
I think we are aligned on this 😄
Great will learn more about Azure batch api
@pditommaso you have documentation on how you worked out gcp or aws implementation?
Hi @vsmalladi
are you trying to develop it? I could support with prereqs but not sure about the nf pieces.
IMHO, sdk docu and https://batch-shipyard.readthedocs.io/en/latest/00-introduction/ should help.
Hi @tbugfinder ,
Yes trying to develop and build it out
@pditommaso you have documentation on how you worked out gcp or aws implementation?
Unfortunately nothing more than the cloud provider docs
I'm working with a group that is interested in this feature, just for maximum portability of our Nextflow pipeline, so I've subscribed!
We are starting to work soon on this. Stay tuned.
I just wanted to add to the thread and say that I would also be very interested in using this feature!
Looks like we've got an implementation to try out, @pditommaso?
https://github.com/nextflow-io/nextflow/tree/master/plugins/nf-azure
The support for Azure Batch is finally available https://twitter.com/nextflowio/status/1364215305359269891
More details at this link https://www.nextflow.io/docs/edge/azure.html
We've been very happy with the release of support for Azure Batch !! I started testing it and got stuck now with the following note:
Install is not supported on dev mode - Missing plugin nf-azure
Even though the plugin is there ... Any ideas?
I'm running with version 21.02.0-edge manually installed, the following command: ./nextflow-21.02.0-edge/launch.sh run nf-core-rnaseq/main.nf -profile test,docker -w az://nf-blob-container/work following guidelines in the article + documentation. So nextflow.config defines plugin id, azure executor, and other azure settings...
Open a separate issue including the command line you are using and the .nextflow.log create by NF