ondemand copies job files to "default directory"

[sync-by-unito]

I have a job directory where required input for job is present , example -

[puneet@cluster1 2]$ pwd
/home/puneet/MyJobs/2
[puneet@cluster1 2]$ ls
a.out  ff.sh  input.dat

I create a new slurm job as -
JobComposer>NewJob>FromSpecifiedPath, and here is what in fill in "Create a new job from a path" form -

Source Path: /home/puneet/MyJobs/2
Name: Myjob2
Script Name: ff.slrm
ClusterName: Cluster1
Account:

on clicking save, a new directory is created -

/home/puneet/ondemand/data/sys/myjobs/projects/default/1/
having contents -
a.out ff.slrm input.dat

Is it possible to run/submit job from /home/puneet/MyJobs/2 via openondemand instead of /home/puneet/ondemand/data/sys/myjobs/projects/default/1/ ?

• concern is that if /home/puneet/MyJobs/2 is large in size (say - more than 100G) then ood may create copy of same.

┆Issue is synchronized with this Asana task by Unito

[sync-by-unito]

➤ Eric Franz commented:

Not yet! I'll take a look and see what type of coding challenge it would be to not copy, but just use the job directory in place. I can't imagine it would be difficult.

You are right if the directory was very large OOD would try to create a copy. I believe we prevent the copying of a directory larger than a gigabyte: https://github.com/OSC/ood-myjobs/blob/d8b1d3fb0dda762f394e649ef837f036bcfd7194/app/models/filesystem.rb#L10

There is a method validate_path_is_copy_safe used to verify it is safe to copy and is used in both places in the app you can initiate a copy of a directory: https://github.com/OSC/ood-myjobs/blob/b694f6d995ca72792ab6b4dc58eba6f62fbdaa55/app/controllers/workflows_controller.rb#L90 and https://github.com/OSC/ood-myjobs/blob/e152e19af1a3285672cf78d1964ee90620a647ef/app/controllers/templates_controller.rb#L43

[sync-by-unito]

➤ puneet336 commented:

Thank you for suggestion - I'll try same.

One thing i can suggest on this is that instead of copying entire directory
/home/puneet/ondemand/data/sys/myjobs/projects/default/1/ could be softlink to /home/puneet/MyJobs/2 and OOD may have a separate option for this type of job creation under "New Job" dropdown.

or contents of /home/puneet/ondemand/data/sys/myjobs/projects/default/1/* could house softlinks to relevant files/directories.

[sync-by-unito]

➤ Strahinja Trecakov commented:

Any updates on this? timeline?

[sync-by-unito]

➤ Eric Franz commented:

trecakov we are discussing about potentially including a fix for this in 1.8 which is slated to be released in July this year.

[sync-by-unito]

➤ Strahinja Trecakov commented:

ericfranz Thanks you