Neutron raytracing done 100% on NVIDIA GPU.
Benchmarked against standard code (VITESS) the raw computation is x18 faster than existing code, but there is also the geometry inversion which makes it crazy fast.
Klaus Habicht and I were standing on the stairs at HMI (now HZB) about 11 years ago, and I was looking between my feet through 3 floors of fire escape. This gave me a funny idea that instead of simulating trajectories you might be able to construct a phase space volume and compress it flat, and the projection operation would do the whole work for you. This ended up as acceptance diagram shading 8 years ago. We also tried GPU back then, but the memory bottleneck was not great. Now there is no memory bottleneck on GPUs, and there is also a decade of GPU development and a lot of GPU memory. So you take the maths from acceptance diagram shading, combine it with monte-carlo, run the whole lot on the GPU.
- Straight pieces of guide
- Curved pieces of guide
- Multichannel benders
- Elliptic curves (polygon)
- Parabolic curves (not tested yet)
- 1D wavelength monitor
- Horizontal phase space map
Monte-Carlo falls under the category of "ridiculously parallelisable" codes. For this reason, using any kind of parallelisation is desirable. In the past, OpenMP and MPI methods have been heavily exploited.
Around a decade ago, I had an intern and we looked at putting neutron ray tracing code on the GPU. GPGPU was fairly new back then, I think CUDA was maybe version 2, and what was awful at that time was the memory bottleneck between system RAM and graphics RAM. It was quicker to run the code on the CPU than wait for the data to go backwards and forwards. So the project stopped, and I wrote NADS instead (https://doi.org/10.1016/j.nima.2009.01.013), which monochromatically rasterised the phase space in two planes to get the kind of speed gains I needed for using metaheuristics to optimise multiple beamlines at the same time (https://doi.org/10.1107/S0021889811013124). This was fine for monochromatic instruments.
Pulsed instrumentation, prevalent at spallation sources, requires a polychromatic approach. Meanwhile, the memory transfer between system RAM and graphics RAM has become very fast, which got me interested in this problem again. There is also enough knowledge gained during the intervening years to do all the work on the GPU in any case, from generating the random trajectories to reducing the arrays in situ, so the RAM transfer speed is irrelevant. Thus we arrive at the code here, as presented at ICANS XXII, Oxford, in March 2017.
Why is it only published on github in 2018? Too much other work was prioritised for us, but today I managed to find time to fix the elliptic geometry bug and the code is finally ready for public digestion. Excellent!
Follow installation instructions in the INSTALL.md file.
An example beamline geometry program will be provided.
The included python script "sandmanPlots.py" is a convenient method for plotting all monitor files and distributions generated by a sandman simulation. It outputs png files in the current working directory.
Functions that have their name ending in "CUDA" are operating directly on the phase space in GPU memory (e.g. moving neutrons down a single section of guide).
Functions that do not have the name "CUDA" are wrapper functions that generate larger geometries from sequences of CUDA operations called multiple times (e.g. lengths of curved guide).
Most of your work will be with wrapper functions, but for simple devices like slits or empty space then you probably will call a CUDA function.
The monitor functions on the CPU are for developers and intended for debugging purposes only and will give incorrect results for users. They immediately dump the phase space present at that position.
The monitor functions running on GPU are for users. These wait until the calculation is finished, then weight the trajectories by the statistical weight and output the phase space in the correct orientation as though the neutrons were simulated in the forwards direction. The GPU functions are the ones that users should be interested in!