Netflix / metaflow

:rocket: Build and manage real-life ML, AI, and data science projects with ease!

Home Page:https://metaflow.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add option to batch decorator to increase ephemeralStorage on Fargate

stev-0 opened this issue · comments

It would be good to have an option to vary the ephemeral storage that a batch job needs on Fargate, since the default (20GB) is relatively low for our needs, and there is no other option on Fargate to increase storage (tmpfs is not an option). Further conversation about this is available on the ask-metaflow chat, but essentially the API is there (was added about a year ago) so it just requires wiring up in metaflow.

I am happy to attempt a PR to do this, I'm not super familiar with the code base but on the other hand I don't think it's a huge job because it's a relatively simple parameter.