OVERVIEW ==================================================================================================================== SurfSuite is a suite of Fortran routines to handle the particle data (positions and velocities) in large cosmological simulations, especially the SURFS simulations produced at ICRAR. In particular, the routines allow the user to (1) sort and quickly retrieve particles from single or multiple snapshot files produced by Gadget-2 (2) rearrage and quickly retrieve/show/analyse all particles in a specific halo identified by VELOCIraptor V0.5 01/03/2017: First release V0.6 18/04/2017: Update, better compatibility with pleiades v0.7 13/02/2019: Significant updates, inclusion of HDF5 module, new showing routine v0.8 18/06/2019: Updated compiler flags, documentation, various fixes v0.9 15/10/2019: Added HDF5 outputs v0.10 15/11/2019: General clean up, improved gethalo routine v0.11 11/02/2020: improved description, cluster optimizations v0.12 21/02/2020: updated parameter files v0.13 23/02/2020: various fixes and clean ups v0.14 24/02/2020: more graphical parameters for showhalo, updated documentation v0.15 25/02/2020: added some file existence checks v0.16 26/02/2020: minor fixes, added a lot of checks v0.17 03/03/2020: improved documentation, generalised snapshot filenames, added trackhalo v0.18 12/03/2020: improved halo tracking, -center changed to positions only v0.19 13/03/2020: added slum example scripts, fixed bug with species in halo tracking v0.20 16/03/2020: added scale factors to tracking outputs v0.21 23/03/2020: additional error messages v0.22 01/04/2020: fixed documentation, centered velocities in showhalo v0.23 07/04/2020: uses new module module_interface to facilitate user interface v0.30 15/04/2020: heavily restructured code using shared modules v0.31 24/04/2020: default parameterset via "parameterset*" v0.32 18/05/2020: updated makefile v0.33 18/05/2020: minor bug fixes v0.34 22/06/2020: minor bug fix in shared modules for argument and HDF5 handling Copyright Danail Obreschkow (danail.obreschkow@icrar.org) QUICK START ON A GENERIC MACHINE ==================================================================================================================== 1) Install gfortran (code tested for GNU Fortran 9.3.0). 2) In a terminal, download surfsuite > git clone https://github.com/obreschkow/surfsuite 3) Go to surfsuite directory > cd surfsuite 4) Modify makefile as needed (e.g. update library paths) and compile the code > make 5) Run test code by typing: > ./surfsuite version 6) Edit the file parameters.txt to point to the correct paths and files 7) Sort GADGET files and use Velociraptor outputs to write particles into halo files: > ./surfsuite makeall 8) Test extracting the information of a single particle > ./surfsuite getparticle 1 9) Test extracting the information of a single group > ./surfsuite gethalo 1 10) Test displaying a halo > ./surfsuite showhalo 1 QUICK START ON ICRAR-HYADES ==================================================================================================================== # install and compile surfsuite in your home directory cd ~ rm -rf surfsuite/ # remove previous surfsuite directory git clone https://github.com/obreschkow/surfsuite # install updated version cd surfsuite # change to surfsuite directory module load gfortran hdf5 # load required modules make system=hyades # compile code ./surfsuite version # check version # assuming that 'makeall' has been run before by another user, you can, for example, # extract particle information for particle number 134 in the SURFS run L210_N1024-Hydro: ./surfsuite getparticle 134 -parameterset L210_N1024-Hydro6D-hyades # NB: it may be necessary to specify the parameter file location using the option "-parameterfile" # you can also extract halo information for halo number 56 in the L210_N1024 run of SURFS ./surfsuite gethalo 56 -parameterset L210_N1024-Hydro6D-hyades # to run a larger task, such as "sortparticles", "makehalos" and "makeall", # use the workload manager SLURM, see https://slurm.schedmd.com/pdfs/summary.pdf ARGUMENTS ==================================================================================================================== Generally, SurfSuite is called as > ./surfsuite task [task_argument] [-option argument] [-option argument] ... The tasks to choose from are + version: returns the version of SurfSuite + simulation: returns basic properties of the Gadget simulation + sortparticles: sorts particles by ID into files of maximun size of 3.6 GB + makehalos: generates binary files, sequentially listing the particles of all halos with species-indices, 3D-positions and 3D-velocities (see file formats below) This uses the halo information from VELOCIraptor. + makeall: runs 'sortparticles' and 'makehalos' + getparticle #particleID: returns species, position and velocity of the specified particle + gethalo #haloID: returns basic properties of the specified halo. Optional arguments: -outputfile: name of a file in which the particles of this halo are saved; if specified the screen output is suppressed -outputformat (default 1): only used if -outputfile given. 1 = hdf5, 2 = ascii, 3 = binary -subhalos (default 0): 0 = do not include particles in subhalos, 1 = include particles in all generations of subhalos -center (default 0): only used if -outputfile given. If set to 1, particle positions are centered to the geometric center (=centre of mass if all species have identical masses) + showhalo #haloID: displays or saves a bitmap-image of the halo. Optional arguments: -outputfile: name of a file in which the image is saved as bitmap (choose *.bmp); if specified the screen output is suppressed -subhalos (default 0): 0 = do not show particles in subhalos, 1 = included particles in all generations of subhalos -at (defaults to the snapshot index specified in the parameter file or via -snapshot): optional index, specifying the snapshot at which the the particles of this halo are to be displayed, allows tracking particles backward and forward in time -mode (default 0): 0 = render particles as points, 1 = show particles as smoothed points, 2 = show small trajectories -npixels (default 800): number of pixels per dimension -sidelength (default 2): sidelength of the displayed images in simulation units -smoothinglength (default 0.2): smoothing length of the displayed images in simulation units; only used if mode=1 -lum (default 1): luminosity scaling factor -gamma (default 0.6): gamma correction value for non-linear brightness scale -projection (default 1): specifies the orthogonal projection, 1 = (x,y), 2 = (y,z), 3 = (z,x) + trackhalo: tracks the particles of a halo back in time and stores the positions and velocities in a hdf5 file. This task requires the sorted Gadget files (produced by the tasks "sortparticles") to exist for all snapshots in the range specified by the arguments "-from" and "-to"; and it requires the halo files (produced by the task "makehalos") to exist for the reference snapshot, i.e. the snapshot specified via the parameter file or -snapshot. Arguments: -outputfile (required): name of a hdf5-file in which the evolving particles of this halo are saved -from (required): index of first snapshot for tracking -to (required): index of last snapshot for tracking -subhalos (default 0): 0 = do not show particles in subhalos, 1 = included particles in all generations of subhalos -center (default 1): if set to 1, particle positions centered to the geometric center of the halo at the default snapshot (specified via the parameter file or -snapshot) Optional arguments used by all routines -parameterfile (default parameters.txt): path+filename of parameter-file -parameterset: if this option is specified, e.g. "-parameterset abc", the parameters listed in this parameterset, e.g. between the lines "parameterset abc" and "end" overwrite the default parameters specified outside the parameterset. The user can mark at most one parameterset in the parameterfile as the default parameterset using an asterix: "parameterset* abc". This parameterset is taken as the default, if the option "-parameterset" is not provided. If no parameterset is marked as default and if no parameterset is selected via "-parameterset", all parameters in parametersets are ignored. -snapshot (default set in parameter-file): overwrites the 'snapshot' number given in parameters.txt to allow a quick access to a different snapshot, keeping all the other parameters identical. -logfile (default none) optional filename of an ascii-file for the screen output -verbose (default n if logfile given, otherwise y): logical flag (y or n) specifying whether the log should be displayed on the screen INPUT FILE FORMATS ==================================================================================================================== Gadget-2 snapshots ------------------ Gadget-2 snapshots must be stored in one or multiple binary files, named [path_gadget]/[snapshot] OR [path_gadget]/[snapshot].0, [path_gadget]/[snapshot].1, ... where bracketed names are specified in the parameter file. In particular, [snapshot] is a character string specified by the three parameters "snapshot" (=snapsnot number), "snapsnhot_prefix" and "snapshot_fmt", where the latter specifies how "snapshot" and "snapshot_prefix" are combined according to Fortran IO rules. E.g. - snapshot 186 - snapshot_ftm (A,I0.4) - snapshot_prefix sn_ generates the filename "sn_0186" The snapshot format must be the standard sequential binary format of Gadget-2 (for n particles): - Header (256 bytes) - Positions (n * 3 * real*4) - Velocities (n * 3 * real*4) - IDs (n * integer*4 or n * integer*8) - [additional properties, which will be ignored] VELOCIraptor files ------------------ When working with VELOCIraptor halos, VELOCIraptor group-files and particles files must be provided, for the Gadget-2 snapshots above: The VELOCIraptor HDF5 groups must be stored in one or several files of name [path_velociraptor]/[snapshot][ext_groups] OR [path_velociraptor]/[snapshot][ext_groups].0, ... The VELOCIraptor HDF5 particles must be stored in one or several files of name [path_velociraptor]/[snapshot][ext_particles] OR [path_velociraptor]/[snapshot][ext_particles].0, ... Scale factor file ----------------- Some task require the scale factors, a = 1/(1+z), corresponding to the snapshots of the simulation. These must be given in an ascii-file, whose filename is specified by the parameter 'file_scalefactors'. The format of this file is ... OUTPUT FILE FORMATS ==================================================================================================================== Sorted Gadget particles ----------------------- The task 'sortparticles' sorts the Gadget particles into new files, named # [path_surfsuite]/[snapshot][ext_sorted] OR # [path_surfsuite]/[snapshot][ext_sorted].0, ... where bracketed names are specified in the parameter file. These files are binary streams, that sequentially list the simulation particles, ordered by their Gadget-ID, in the format - id(1),species(1),x(1),y(1),z(1),vx(1),vy(1),vz(1) - id(2),species(2),x(2),y(2),z(2),vx(2),vy(2),vz(2) ... where id(i) is an 8-byte integer, species(i) is a 4-byte integer and the other 6 variables are 4-byte reals. Thus each particle takes exactly 36 bytes. Each file contains up to 10^8 particles, such that the file sizes are limited to about 3.6GB. Whether one or several files are used depends on the number of particles. NB: The maximum number of particles per file can be changed programmatically, via a the global parameter nparticles_per_sorted_file in the module 'module_global.f95'. This sorting is just by Gadget particle id. It has nothing to do with VELOCIraptor and can be run without available VELOCIraptor data. Once the particles have been sorted with 'sortparticles', the task 'getparticle' can be used to retrieve the particle information. Particles sorted into haloes ---------------------------- If VELOCIraptor files (see above) are available, the task 'makehalos' can be used to sort the Gadget particles sequentially halo-after-halo into one or several files, named [path_surfsuite]/[snapshot][ext_halos] OR [path_surfsuite]/[snapshot][ext_halos].0, ... The number of such files is identical to the number of VELOCIraptor group files. Each new file is a binary stream that sequentially lists particles in the format - id(1),species(1),x(1),y(1),z(1),vx(1),vy(1),vz(1) - id(2),species(2),x(2),y(2),z(2),vx(2),vy(2),vz(2) ... with identical variable types to those above. Simultaneously, the task 'makehalos' also generates a single master file, named [path_surfsuite]/[snapshot][ext_halolist] with pointers to the individual halos in the files [path_surfsuite]/[snapshot][ext_halos]. This file is a binary stream of the format - idfile(1), position(1), npart(1), npartsub(1), parentid(1), nchildren(1), firstchildid(1), siblingid(1) - idfile(2), position(2), npart(2), npartsub(2), parentid(2), nchildren(2), firstchildid(2), siblingid(2) ... All 8 variables are 4-byte integer, thus the full stream has exactly nhalos*32 bytes. The meaning of these 8 variables is: index of the halo-file, in which to look for the particles; position (in bytes) of the first particle; number of particles in the halo without substructure; number of particles in substructure only; halo ID of the parent halo, -1 is used for 1st generation halos; number of subhaloes in the generation immediately below; halo index of the first subhalo, -1 if no subhaloes; halo index of the next subhalo inside the same parent halo, -1 if none. Once the particles have been sorted into haloes with 'makehalos', the tasks 'gethalo' and 'showhalo' can be used. Auxiliary analysis path ----------------------- surfsuite automatically generates the empty directory [path_analysis], specified in the parameter file. This empty directory is meant to serve for post-processing purposes by other code that can also access this parameter file, such as the tool 'surfanalysis' (not public). PACKAGED FILES ==================================================================================================================== + 'README' = this file + 'surfsuite.f95' main program + 'module_xxx.f95' various modules with subroutines + 'parameters.txt' sets paths and file name extensions + 'makefile' contains the compiling instructions for the gfortran compiler.