liuheng92 / setup

Setup a new machine without sudo!

Setup a New Machine without sudo!

Download, compile and install commonly used software to a custom path.

Term of Usage

• Star the repo before clone
• File issue if it does not work
• Unstar if you feel it is unmaintained

Usage

# configure setup path, you can put them into your `.bashrc` or `.zshrc`
export ZZROOT=$HOME/app
export PATH=$ZZROOT/bin:$PATH
export LD_LIBRARY_PATH=$ZZROOT/lib:$ZZROOT/lib64:$LD_LIBRARY_PATH

# e.g. install git
sh zzgit.sh

jlinstall

jlinstall is another way to install softwares. It uses pre-built binaries so that no compilation is needed. The default install path of jlinstall is JLROOT=$ZZROOT/jl.

export ZZROOT=$HOME/app
export JLROOT=$ZZROOT/jl
export PATH=$ZZROOT/bin:$JLROOT/bin:$PATH
export LD_LIBRARY_PATH=$ZZROOT/lib:$ZZROOT/lib64:$JLROOT/lib:$JLROOT/lib64:$LD_LIBRARY_PATH

NOTE: If the download speed of the machine is too slow, you can put the git.tar.gz (using git as an example) into $ZZROOT/downloads before running the script. For the download link, check the DOWNLOADURL variable in its script; for the exact name of the software, check the NAME variable.

NOTE2: If you compile from source, please make sure that they are compiled using the same gcc version!

Supported Softwares

Verified on Ubuntu 18.04

Updated: July 8, 2020

libs

• boost (1.71.0)
  • install script: ./zzboost.sh
  • post install: export BOOST_ROOT=$ZZROOT
• gmp (6.1.2)
  • install script: ./zzgmp.sh
• mpc (1.1.0)
  • depends on: gmp, mpfr
  • install script: ./zzmpc.sh
• mpfr (4.0.2)
  • depends on: gmp
  • install script: ./zzmpfr.sh
• nasm (2.14.02)
  • install script: ./zznasm.sh
• yasm (1.3.0)
  • install script: ./zzyasm.sh
• ncurses (6.1)
  • install script: ./zzncurses.sh
  • post install: export TERMINFO=$ZZROOT/share/terminfo
• freetype (2.10.1)
  • install script: ./zzfreetype.sh
• opencv (4.1.2)
  • note: this config is designed for denseflow. Please modify the cmake flags for your needs.
  • depends on: ccache, ffmpeg
  • install script: ./zzopencv.sh
  • post install: export OpenCV_DIR=$ZZROOT
• openssl (1.1.1d)
  • note: both PREFIX and OPENSSLDIR are set to $ZZROOT/ssl
  • install script: ./zzopenssl.sh
  • post install: for cmake export OPENSSL_ROOT_DIR=$ZZROOT/ssl, and for configure --with-ssl=$ZZROOT/ssl
• libcurl (7.67.0)
  • depends on: autoconf, automake, libtool, m4, nroff, perl, openssl
  • install script: ./zzlibcurl.sh
• libevent (2.1.11)
  • install script: ./zzlibevent.sh
• libvpx (master)
  • install script: ./zzlibvpx.sh
• libx264 (master)
  • install script: ./zzlibx264.sh
• libx265 (3.2.1)
  • install script: ./zzlibx265.sh
• libz (1.2.11)
  • install script: ./zzlibz.sh
• libzstd (1.4.4)
  • install script: ./zzlibzstd.sh
• libzip (1.5.2)
  • install script: ./zzlibzip.sh
• libtool (2.4.6)
  • install script: ./zzlibtool.sh
• libjpeg (6b)
  • depends on: libtool
  • install script: ./zzlibjpeg.sh
• libjpeg-turbo (2.0.3)
  • depends on: nasm, yasm
  • install script: ./zzlibjpeg-turbo.sh
• libpng (1.6.37)
  • install script: ./zzlibpng.sh
• libtiff (4.1.0)
  • install script: ./zzlibtiff.sh
• libwebp (1.0.3)
  • install script: ./zzlibwebp.sh
• giflib (5.2.1)
  • install script: ./zzgiflib.sh
• libopenjp2 (2.3.1)
  • install script: ./zzlibopenjp2.sh

tools

• autoconf (2.69)
  • depends on: m4, perl
  • install script: ./zzautoconf.sh
• ccache (3.7.6)
  • install script: ./zzccache.sh
  • post install: customize cache dir by export CCACHE_DIR=~/.ccache to avoid permission issue
• cmake (3.16.0)
  • install script: ./zzcmake.sh
• anaconda3 (2019.10)
  • note: anaconda is installed default on $HOME/anaconda3
  • install script: ./zzconda.sh
  • post install: export PATH=$HOME/anaconda3/bin:$PATH
• ffmpeg (4.2.1)
  • depends on: nasm, yasm, libx264, libx265, libvpx
  • install script: ./zzffmpeg.sh
• gcc (7.5.0)
  • note: CUDA 10 support up to gcc 7, CUDA 9 support up to gcc 5.
  • depends on: gmp, mpfr, mpc
  • install script: ./zzgcc.sh
• git (2.24.0)
  • depends on: libcurl, libz
  • install script: ./zzgit.sh
• hdf5 (1.10.5)
  • install script: ./zzhdf5.sh
• htop (2.2.0)
  • depends on: ncurses
  • install script: ./zzhtop.sh
• julia (1.3.0)
  • install script: ./zzjulia.sh
• autojump (master)
  • install script: ./zzautojump.sh
  • post install: put lines of configs in .zshrc
• imagemagick (latest)
  • depends on: freetype, libjpeg-turbo, libpng, perl
  • install script: ./zzimagemagick.sh
• nmon (16j)
  • depends on: ncurses
  • install script: ./zznmon.sh
• zsh (5.7.1)
  • depends on: ncurses
  • install script: ./zzzsh.sh
  • post install: to set zsh as default shell without root, add exec $ZZROOT/bin/zsh -l to .bashrc (example)
• ohmyzsh (latest)
  • depends on: zsh
  • install script: ./zzohmyzsh.sh
• perl (5.30.1)
  • install script: ./zzperl.sh
• tmux (3.0a)
  • depends on: libevent
  • install script: ./zztmux.sh
• unrar (5.8.4)
  • install script: ./zzunrar.sh
• vmtouch (1.3.1)
  • install script: ./zzvmtouch.sh

zz scripts & customized tools

• sshpass (support 2-step verification)
  • depends on: oathtool, autoconf
  • install script: ./zzsshpass.sh
• denseflow (extract frames, optical flow, resize, and more!)
  • depends on: boost, opencv, cmake
  • install script: ./zzdenseflow.sh
• wslssh.bat (use ssh of wsl in win10, no installation, .bat script as is)
  • depends on: sshpass
  • you can set vscode ssh path to this wslssh.bat, work together with sshpass (installed in wsl)
• sshkey.sh (gen an ssh key)
  • run script: ./sshkey.sh
• zshplugin.sh (completions, autosuggestions, syntax-highlighting)
  • depends on: git
  • install script: ./zshplugin.sh
  • post install: put lines of configs in .zshrc
• nordtmux.sh (nord theme for tmux)
  • depends on: git, tmux
  • install script: ./nordtmux.sh
• dircolor.sh (nord theme for zsh)
  • install script: ./dircolor.sh
  • post install: put lines of configs in .zshrc

zz configures

• .tmux.conf (tmux configuration)
• .zshrc (zsh configuration)
• .netrc (lftp password-less configuration)
• .ssh/config (suppress git clone warnings)
• .lftp/rc (suppress lftp ssl warning)

misc

• opengl on remote machine
• useful Julia packages: ArgParse Glob Images FileIO StatsBase Shell UnicodePlots PyCall PyPlot Plots ImageFiltering ImageMagick Interpolations Revise OhMyREPL DataFrames ProgressMeter JLD2 Distances ImageTransformations JuMP HDF5 ImageInTerminal ImageView DataStructures JSON JuliaFormatter
• free JuMP solvers: Cbc Clp COSMO CSDP ECOS GLPK Juniper OSQP ProxSDP SCS SDPA
• good reference: http://www.linuxfromscratch.org/lfs/view/development/index.html and http://www.linuxfromscratch.org/blfs/view/svn/index.html and https://pkgs.org/
• CUDA arch and CUDA gencode list
• Instruction on how to load jpeg faster. https://github.com/innerlee/cvbenchmark#setup
• View image in terminal: viu and tiv
• image lossless optimization: jpegtran optipng
• image lossy optimization: pngquant
• ffmpeg tricks
  • silent: ffmpeg -hide_banner -loglevel panic
  • get resolution ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=p=0 input.mp4
  • count frames ffmpeg -i input.mp4 -map 0:v:0 -c copy -f null - 2>&1 | grep frame
  • remove duplicated frames ffmpeg -i input.mp4 -vf mpdecimate -vsync vfr out.mp4

faq

• Q: My boost segfaluts :(
  A: Make sure they the boost are comipiled and linked using the same gcc/g++ version

• Q: Cert verify failed: BADCERT_NOT_TRUSTED :(
  A: Try export SSL_CERT_DIR=/etc/ssl/certs

• Q: warning: templates not found in /workspace/destdir/share/git-core/templates :(
  A: Add the following to $HOME/.gitconfig

[init]
        templatedir = $HOME/app/share/git-core/templates

• Q: fatal: unable to find remote helper for 'https' :(
  A: Add the following to $HOME/.zshrc (or .bashrc of course)

export GIT_EXEC_PATH=$ZZROOT/libexec/git-core
export GIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crt

• Q: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications. :(
  A: Add the following to $HOME/.zshrc (or .bashrc of course)

export GIO_EXTRA_MODULES=/usr/lib/x86_64-linux-gnu/gio/modules/

• Q: import pyvips segfault :(
  A: Make sure gstreamer-orc <0.4.30, imagemagick <7.0.9.

conda install -c conda-forge imagemagick=7.0.8
conda install -c conda-forge gstreamer-orc=0.4.29

• Q: GitError(Code:ECERTIFICATE, Class:SSL, the SSL certificate is invalid: 0x08 - The certificate is not correctly signed by the trusted CA) :(
  A: Run the following

julia> import LibGit2
julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs/ca-certificates.crt")

• Q: dotnet error :(
  A: Add the following to $HOME/.zshrc (or .bashrc of course)

export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true
export MSBuildSDKsPath=$DOTNET_ROOT/sdk/3.1.102/Sdks

• Q: Install OpenCV: CUDA Version wrong:(
  A: Add the path of the correct version CUDA to zzopencv.sh when runing cmake

cmake   ......
        -DCUDA_TOOLKIT_ROOT_DIR=/mnt/lustre/share/cuda-9.0/ \
        ......

• Q: Install Denseflow: OpenCV Version wrong, can't find customized path for opencv:(
  A: Add the path of the correct version opencv and corresponding cuda version to zzdenseflow.sh when runing cmake

cmake -DCMAKE_INSTALL_PREFIX=$ROOTDIR \
      -DOpenCV_DIR=$ROOT_DIR/lib64/cmake/opencv4 \
      -gencode=arch=compute_61,code=sm_61 \
      -DCUDA_TOOLKIT_ROOT_DIR=/mnt/lustre/share/cuda-9.0/ ..

• Q: Unzip filenames looks messy when there are unicodes :(
  A: Specify the encoding of filenames.

$ unzip -h
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.
  ...
  -O CHARSET  specify a character encoding for DOS, Windows and OS/2 archives
  -I CHARSET  specify a character encoding for UNIX and other archives
  ...

unzip -O GB18030 gb18030.zip -d target_dir

• Q: Get locale warnings :(
  A: Run

export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
locale-gen en_US.UTF-8
sudo dpkg-reconfigure locales

• Q: moov atom not found :(
  A: Use https://github.com/anthwlock/untrunc to fix it.