# -*- Mode: sh; sh-basic-offset:2 ; indent-tabs-mode:nil -*- # # Copyright (c) 2014-2017 Los Alamos National Security, LLC. All rights # reserved. # $COPYRIGHT$ # # Additional copyrights may follow # # $HEADER$ # HIO Readme ========== Last updated 2017-01-12 libHIO is a flexible, high-performance parallel IO package developed at LANL. libHIO supports IO to either a conventional PFS or to DataWarp with management of DataWarp space and stage-in and stage-out from and to the PFS. libHIO has been released as open source and is available at: https://github.com/hpc/libhio For more information on using libHIO, see the github package, in particular: README libhio_api.pdf hio_example.c README.datawarp See file NEWS for a description of changes to hio. Note that this README refers to various LANL clusters that have been used for testing HIO. Using HIO in other environments may require some adjustments. Building -------- HIO builds via a standard autoconf/automake build. So, to build: 1) Untar 2) cd to root of tarball 3) module load needed compiler or MPI environment 4) ./configure 5) make Additional generally useful make targets include clean and docs. make docs will build the HIO API document, but it requires doxygen and various latex packages to run, so you may prefer to use the document distributed in file design/libhio_api.pdf. Our target build environments include gcc with OpenMPI on Mac OS for unit test and gcc, Intel and Cray compilers on LANL Cray systems and TOSS clusters with Cray MPI or OpenMPI. Included with HIO is a build script named hiobuild. It will perform all of the above steps in one invocation. The HIO development team uses it to launch builds on remote systems. You may find it useful; a typical invocation might look like: ./hiobuild -c -s PrgEnv-intel,PrgEnv-gnu hiobuild will also create a small script named hiobuild.modules.bash that can be sourced to recreate the module environment used for build. API Example ----------- The HIO distribution contains a sample program, test/hio_example.c. This is built along with libHIO. The script hio_example.sh will run the sample program. Simple DataWarp Test Job ------------------------ The HIO source contains a script test/dw_simple_sub.sh that will submit a simple, small scale test job on a system with Moab/DataWarp integration. See the comments in the file for instructions and a more detailed description. Testing ------- HIO's tests are in the test subdirectory. There is a simple API test named test01 which can also serve as a coding example. Additionally, other tests are named run02, run03, etc. Theses tests are able to run in a variety of environments: 1) On Mac OS for unit testing 2) On a non-DataWarp cluster in interactive or batch mode 3) On one of the Trinity systems with DataWarp in interactive or batch mode run02 and run03 are N-N and N-1 tests (respectively). Options help can be displayed by invoking with a -h option. These tests use a common script named run_setup to process options and establish the testing environment. They invoke hio using a program named xexec which is driven by command strings contained in the runxx test scripts. A typical usage to submit a test DataWarp batch job on the small LANL test system named buffy might look like: cd <tarball>/test ./run02 -s m -r 32 -n 2 -b Options used: -s m ---> Size medium (200 MB per rank) -r 32 ---> Use 32 ranks -n 2 ---> Use 2 nodes -b ---> Submit a batch job The runxx tests will use the hiobuild.modules.bash files saved by hiobuild (if available) to reestablish the same module environment used at build time. A multi-job submission script to facilitate running a large number of tests with one command is available. A typical usage for a fairly thorough test on a large system like Trinity might look like: run_combo -t ./run02 ./run03 ./run12 -s x y z -n 32 64 128 256 512 1024 -p 32 -b This will submit 54 jobs (3 x 3 x 6) with all combinations of the specified tests and parameters. The job scripts and output will be in the test/run subdirectory. Step by step procedure for building and running HIO tests on LANL system Trinite: --------------------------------------------------------------------------------- This procedure is accurate as of 2017-01-12 with HIO.1.3.0.6. 1) Get the distribution tarball libhio-1.3.0.6.tar.gz from github at https://github.com/hpc/libhio/releases 2) Untar 3) cd <dir>/libhio-1.3 ( <dir> is where you untarred HIO ) 4) ./hiobuild -cf -s PrgEnv-intel,PrgEnv-gnu At the end of the build you will see: tt-fey1 ====[HIOBUILD_RESULT_START]===()=========================================== tt-fey1 hiobuild : Checking /users/cornell/tmp/libhio-1.3/hiobuild.out for build problems 24:configure: WARNING: using cross tools not prefixed with host triplet 259:Warning: tt-fey1 hiobuild : Checking for build target files tt-fey1 hiobuild : Build errors found, see above. tt-fey1 ====[HIOBUILD_RESULT_END]===()============================================= Ideally, the two warning messages would not be present, but at the moment, they can be ignored. 5) cd test 6) ./run_combo -t ./run02 ./run03 ./run12 ./run20 -s s m -n 1 2 4 -p 32 -b This will create 24 job scripts in the libhio-1.3/test/run directory and submit the jobs. Msub messages are in the corresponding .jobid files in the same directory. Job output is directed to corresponding .out files. The number and mix of jobs is controlled by the parameters. Issue run_combo -h for more information. 7) After the jobs complete, issue the following: grep -c "RESULT: SUCCESS" run/*.out If all jobs ran OK, grep should show 24 files with a count of 1. Like this: cornell@tr-login1:~/pgm/hio/tr-gnu/libhio-1.3/test> grep -c "RESULT: SUCCESS" run/*.out run/job.20170108.080917.out:1 run/job.20170108.080927.out:1 run/job.20170108.080936.out:1 run/job.20170108.081422.out:1 . . . . run/job.20170108.082133.out:1 run/job.20170108.082141.out:1 Investigate any missing job output or counts of 0. 8) Alternatively, cd to the libhio-1.3/test directory, run the script ./check_test This will show how many jobs are queued and currently running and how many output files are incomplete or have failures. 9) Resources for better understanding and/or modifying these procedures: libhio-1.3/README libhio-1.3/README.datawarp libhio-1.3/hiobuild -h libhio-1.3/test/run_combo -h libhio-1.3/test/run_setup -h libhio-1.3/test/run02, run03, run12, run20 libhio-1.3/test/xexec -h libhio-1.3/design/libhio_api.pdf libhio-1.3/test/hio_example.c 10) Additional test commands, check the results the same way as above. Very simple small single job Moab/DataWarp test: ./run02 -s t -n 1 -r 1 -b Alternate multi job test suitable for a large system like Trinity: ./run_combo -t ./run02 ./run03 ./run12 ./run20 -s l x -n 1024 512 256 128 64 -p 32 -b Additional many job submission contention test ./run90 -p 5 -s t -n 1 -b This test submits two jobs that each submit two additional jobs. Job submission continues until the -p parameter is exhausted. So, the total number of jobs is given by (p^2) - 2. Be cautious about increasing the -p parameter. Since this is only a job submission test, the normal scan for RESULT: SUCCESS is not applicable. Simply wait for the queue to empty and look for the expected number of .sh and .out files in the run directory. If there are any .sh files without corresponding .out files, look for errors via checkjob -v on the job IDs in the .jobid file. DataWarp stage-out can impose a significant load on the scratch file system. To inhibit stage-out (which will reduce test coverage) set the environment: export HIO_datawarp_stage_mode=disable. --- End of README ---