- Author
James McIntosh
- Version
0.0.3
I was getting tired of writing custom submission scripts for every project I was working on. These are basic scripts to automatically submit multiple parallel jobs to Columbia's Habanero HPC. Includes various temporary directory settings to try to make the parallelisation more stable. (Each folder has submission scripts for a different project). Should work with other SLURM systems but paths may require tweaking. Tested mainly on Ubuntu 18.04, although should now work on Windows systems too.
Other info ======= https://wikis.cuit.columbia.edu/confluence/display/rcs/Habanero+-+Job+Examples https://github.com/gaoyuanjun/hpc_tutorial
Quick Submit ======= Archives the folder you specify, sends it to the cluster and runs the function you specify within that folder.
Makes use of SSH/SFTP/SCP For Matlab (v2) to connect to the cluster. This should be downloaded and placed in the auxf directory. Also uses (already in auxf) findobj so that the passwords that get used don't stay in Matlab history (definitely not guaranteed). It's also possible to load the password from a .mat file stored on disk (which would usually be stored encrypted).
Example use:
matdir = '/hdd/Cloud/project1';
matname = 'project1';%project name
d.n_par = 24;
d.walltime = '03:00:00';
matfunc = "function_name_to_run";%runs function_name_to_run.m
hab_submit(columbia_id,account_type,matfunc,matdir,matname,'n_par',d.n_par,'walltime',d.walltime);Will move matdir to the cluster after archiving into a folder on Habanero named project1. Will then run "function_name_to_run" from within project after starting the parallel pool.
If project1 results are being written (for example) to folder 'sim' in 'project1', then sim folder can be retrieved like so:
hab_recover(columbia_id,account_type,'sim','project1');