A re-write of the C/C++ FITSWebQL in Rust. The previous C/C++ version can still be found here: http://jvo.nao.ac.jp/~chris/fitswebql.html
The Rust version is now in a maintenance mode. A new hybrid C & FORTRAN cluster edition - scaling to over 1TB-large files - is in development: https://github.com/jvo203/FITSWEBQLSE
make sure the git tool is installed on your system:
https://git-scm.com/book/en/v2/Getting-Started-Installing-Git
download a stable release of fits_web_ql from
https://github.com/jvo203/fits_web_ql/releases
or alternatively clone the latest development version fits_web_ql onto your computer with the git tool:
cd <your_projects_folder>
git clone https://github.com/jvo203/fits_web_ql.git
after cloning the fits_web_ql repository the 809MB-large spectral lines database needs to be downloaded from
http://jvo.nao.ac.jp/~chris/splatalogue_v3.db
and placed inside the fits_web_ql directory
(for example "wget http://jvo.nao.ac.jp/~chris/splatalogue_v3.db")
First and foremost the Rust language version 1.31.0 or higher (Rust 2018 edition) must be installed:
macOS:
brew install rust
other platforms:
make and other command-line software development tools
Ubuntu Linux: open a terminal and type
sudo apt-get install build-essential
macOS: from the command-line
xcode-select --install
then install the Homebrew package manager:
execute the following from the command line (you will be prompted for your password in order to complete the installation):
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install cmake
install a free open-source Intel SPMD Program Compiler (ispc)
macOS:
brew install ispc
other platforms:
in the "Downloads" section select the binary corresponding to your platform
then place the extracted executable "ispc" in your PATH (for example ~/bin/ or /usr/local/bin)
install clang
macOS: this step can probably be skipped as clang should already be present, if not you should run "xcode-select --install" from the command-line to install the software development environment
on macOS llvm-config is needed too:
brew install llvm
followed by
echo 'export PATH="/usr/local/opt/llvm/bin:$PATH"' >> ~/.zshrc
Ubuntu Linux:
sudo apt install clang llvm
CentOS:
sudo dnf install llvm-devel clang-devel
install nasm and yasm assembler compilers
macOS:
brew install nasm yasm
Ubuntu Linux:
sudo apt-get install nasm yasm
install a libyuv library (YUV rescaling/image inversion):
(macOS libyuv compilation problem, for the solution see: https://github.com/lemenkov/libyuv/issues/18)
git clone https://github.com/lemenkov/libyuv
cd libyuv
mkdir -p build
cd build
cmake -DCMAKE_POSITION_INDEPENDENT_CODE=ON ..
make
sudo make install
install Google's libvpx 1.11.0 or higher; if your libvpx is < 1.11 fits_web_ql version 4.2.11 should be used
macOS: "brew install libvpx"
other systems follow:
http://www.linuxfromscratch.org/blfs/view/svn/multimedia/libvpx.html
cd libvpx
./configure --enable-pic
make
sudo make install
(when compiling from source enforce -fPIC by means of the configure flag --enable-pic)
install x265 version 2.8 or higher
macOS: "brew install x265"
other systems follow:
http://www.linuxfromscratch.org/blfs/view/svn/multimedia/x265.html
cd x265_2.9
mkdir -p build
cd build
cmake ../source
make
sudo make install
please be sure to have nasm installed beforehand when building from source, plus NUMA API: numactl and numa development library libnuma (package libnuma-dev on Ubuntu)
install sqlite3
macOS: normally sqlite3 comes pre-installed in macOS, if not you may install it manually with "brew install sqlite3"
Ubuntu Linux: "sudo apt install libsqlite3-dev"
some Linux systems, for example Ubuntu, CentOS 6 and 7, need the following environment variables to be set before running fits_web_ql:
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
it is best to append these lines into your .bashrc
on macOS (especially macOS Ventura 13.2 and higher) the following environment variable needs to be set (for example in your .zshrc):
export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/usr/local/lib
cd into the fits_web_ql directory and execute
cargo run --release
after a successful compilation (it may take some time!) point your web browser to http://localhost:8080
press CTRL+C to exit the program
to avoid needless compilation before running one can build it once with "cargo build --release" and then launch it many times by executing
target/release/fits_web_ql
or
target/release/fits_web_ql --port 8080 --interface 0.0.0.0 --home /a/path/to/your/FITS/mount
cd into the fits_web_ql directory and execute
cargo run --features 'jvo production cdn zfp' --release
or
cargo run --features 'jvo production cdn zfp' --release -- --path fitswebql_v4
The "--features" option enables extra functionality. JVO-reserved features are "jvo" and "production". "cdn" can be used by anyone to speed up delivery of static resources by utilising a jsDelivr open-source content delivery network (https://www.jsdelivr.com). "cdn" is especially recommended if many users are accessing a remote FITSWebQL server. There is no need to use it on your personal computer. "zfp" enables ZFP compression for FITS data cubes held in an internal FITSWebQL cache in order to (theoretically) speed-up loading times (see a note at the end of this README). "ipp" enables use of the Intel Integrated Performance Primitives (IPP) library in some places (for example rescaling images/videos).
an alternative HTTP port
cargo run --features 'cdn' --release -- --port 8000
an alternative URL path (JVO-specific)
cargo run --features 'jvo production cdn zfp' --release -- --path fitswebql_v4
an alternative network interface (only needed to make the local version operate in a remote-server mode)
cargo run --release -- --interface 0.0.0.0
an alternative home directory (FITS data storage)
cargo run --release -- --home /a/path/to/your/FITS/mount
combined options
cargo run --features 'cdn' --release -- --port 8000 --interface 0.0.0.0 --home /a/path/to/your/FITS/mount
or
cargo build --features 'cdn' --release
target/release/fits_web_ql --port 8000 --interface 0.0.0.0 --home /a/path/to/your/FITS/mount
FITSCACHE placement
A hint how to speed-up FITSWebQL when using SSDs. It it best to clone fits_web_ql onto a directory residing on the fastest storage medium you have (ideally NVME SSD). Inside fits_web_ql there is an internal FITSCACHE directory where the program caches half-float-converted FITS files (applicable only to bitpix = -32).
The first time a FITS file is accessed it will be read from its original location (ideally from a fast NVME SSD). The second time somebody accesses that same FITS file, fits_web_ql will read the FITS header from the original location and then proceed to load half-float binary cache from the FITSCACHE directory.
So even if your large FITS files reside on a fast SSD, if fits_web_ql itself is located on a slow HDD second-time loads of FITS files will appear slow compared with first-time accesses.
One can also experiment with symbolic links to a separate FITSCACHE directory residing on a fast storage medium.
enable the "zfp" feature
i.e. cargo run --features 'zfp' --release
This feature replaces the half-float storage with ZFP compression (https://github.com/LLNL/zfp), which speeds-up second-time loads on multi-core systems. The HDD/SSD cache storage uses zfp 2d arrays instead of half-floats, which are converted to a half-float RAM storage upon loading. Decompressing data is very CPU intensive, hence this feature is only recommended if your server contains a sufficient number of CPU cores (i.e. >= than #memory channels). Otherwise speed savings from reading smaller file sizes will be eaten-up by increased CPU times of decompressing the data. cmake3 needs to be installed on your system.
A personal comment: ZFP compresses data using 4x4 blocks which introduces undesirable blocky artifacts at too high compression ratios. Compression/decompression speeds are not fast. In author's experience, compressing FITS data cubes with Radial Basis Functions (a terribly slow process by itself!) results in much better compression ratios with no visible artifacts. The author will continue experimenting with various compression methods (including wavelets).
enable use of Intel IPP via an experimental feature "ipp"
i.e. cargo run --features 'zfp ipp' --release
The Intel Integrated Performance Primitives (IPP) library can be obtained free of charge from https://software.intel.com/en-us/intel-ipp
IMPORTANT: please make sure that the IPPROOT environment variable is set following the IPP installation instructions (i.e. echo $IPPROOT /opt/intel/compilers_and_libraries_2019.4.243/linux/ipp)