Setting up macOS for scientific computing and software development

Author: Matthias Cuntz, mc (at) macu (dot) de
Modified: 09.01.2024

Inspired by guides of Dirk Avery, Vinicius De Antoni, Sourabh Bajaj, Nicolas Hery.

This is a reminder for myself how to setup a new Mac or updating the OS, which might be useful for others. The repository includes some of my dot-files as a reference.

How to update the OS
- Fresh install
- Update via /Software Update/
Setup macOS

How to update the OS

There are basically two options:

Reformat the startup disk and install a fresh new system, or
Use Software Update to update macOS on top of the existing OS.

In both cases, a little bit of preparation helps.

Fresh install

If you’re using a Mac with an Intel processor and the Apple T2 Security Chip, make sure that Startup Security Utility is set to allow booting from external or removable media. Follow the instructions on support.apple.com. You can find the list of models with the T2 Security chip at https://support.apple.com/en-au/103265 or under System Information / > /Hardware > Controller.

Before starting

You might want to use iCloud for your Contacts, Calendars, Reminders, Notes, and Keychain. If you had not switched it on, you should do this a couple of hours before installation because it takes some time until things get synchronised. Some logoff/login might also help. You can check all but Keychain on icloud.com. You can switch it off again on the updated system after synchronisation finished, if not wanted.
Backup all your files including Documents, Movies, Pictures. Check the Desktop as well.
Also store all your dot-files and dot-directories from the ${HOME} directory such as .bash_profile, .bashrc and .ssh.
You might want to write down your Computer Name from System Preferences > Sharing (macOS <= 12) or System Settings > General > Sharing > Local hostname (macOS >= 13).
You might also like to write down your username. Type in the terminal:
```
whoami
    
```
A list of your installed programs in Applications might come in handy.

Make a bootable installer for macOS on USB

From support.apple.com:

Download a macOS installer such as macOS Sonoma from the App Store. When the macOS installer opens, quit it without continuing installation.
The installer should have a size of about 12-16 GB. Based on your Mac, only a much smaller app might have been downloaded, which would later download the actual files during installation. This does not allow to create a bootable thumb drive. Try to get the full installer for example from the Apple Developer site.
Format a USB flash drive of at least 16 GB storage as Mac OS Extended with name InstallSonoma, for example.

In Terminal type:

	 sudo /Applications/Install\ macOS\ Sonoma.app/Contents/Resources/createinstallmedia --volume /Volumes/InstallSonoma --nointeraction

Installing macOS with the bootable installer

Basically follow support.apple.com because the steps are slightly different between Intel and Apple Silicon chips.

Startup is different if you have a Mac with Apple silicon chips or Intel processors.
a) Intel processor (check the notes at the beginning of the section about Apple’s T2 Security Chip)
- Shut down.

Plug the bootable installer into a Mac that is connected to the internet.
Turn on your Mac, then immediately press and hold the Option (Alt) key. Release the Option key when you see a dark screen showing your bootable volumes.
Select the volume containing the bootable installer, i.e. Install macOS Sonoma, and press Return.
Choose your language, if prompted.
Select Disk Utility. Choose the Internal disk, e.g. Macintosh HD. Erase it using the Format APFS. Use the button Erase Volume Group instead of Erase if available; otherwise use Erase.
- Select any other internal volumes in the sidebar, then click the delete volume (–) button in the toolbar to delete that volume. Disregard any internal volume named Macintosh HD or Macintosh HD – Data, as well as any volumes in the External and Disk Images sections of the sidebar.
- Then quit Disk Utility.
- Select Install macOS Sonoma, click Continue and follow the on-screen steps.
a) Apple silicon
- Shut down.
- Plug the bootable installer into a Mac that is connected to the internet.
Turn on the Mac and continue to hold the power button until you see the startup options window, which shows your bootable volumes.
Select the volume containing the bootable installer, i.e. Install macOS Sonoma, then click Continue.

Update via Software Update

Before starting

It is always recommended to make a backup of your system before an upgrade.
Update all your installed apps in Applications. The newest app versions will probably already be suitable for the new macOS version. It avoids the problem that you cannot open the old app version on the new macOS system anymore to do the simple update and then have to uninstall/install the app again, which might also need the licence key, etc.
For this, open each non-Apple application and Check for Updates….

Use the App Store… for all Apple programs and apps downloaded from the App Store..
The Software Update takes care of all your accounts, passwords, etc. The step before takes care of all the installed applications. But anything installed from the command line will (most probably) not work anymore. So I first uninstall the all things that will be installed in the steps below, which are Homebrew, LaTeX, Python/pyenv, and everything installed with install_netcdf. I also set back Apple’s bash shell as default (because the newer bash shell is removed together with Homebrew).
Set Apple’s bash shell as default:
```
chsh -s /bin/bash
    
```
You have to open a new login terminal so that it takes effect.

Uninstall panoply:
```
brew uninstall panoply
    
```
```
brew uninstall temurin
    
```
```
brew uninstall emacs
    
```
Remove Homebrew:
```
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/uninstall.sh)"
    
```
```
[[ -f ${HOME}/.fzf.bash ]] && rm -f ${HOME}/.fzf.bash
    
```
```
[[ -f ${HOME}/.fzf.zsh ]] && rm -f ${HOME}/.fzf.zsh
    
```
```
[[ -d /opt/homebrew ]] && sudo rm -r /opt/homebrew
    
```
Remove LaTeX:
```
sudo rm -r /usr/local/texlive/
    
```
```
sudo rm -r /Library/TeX/
    
```
Remove pyenv’s Python versions:
```
rm -rf ${HOME}/.pyenv
    
```
Remove everything under /usr/local. This seems extreme but Apple has nothing installed under /usr/local, so it comes all from you or Homebrew. You might change into /usr/local and remove selected directories by hand. For example, our virus scanner at work installs its uninstall-script in /usr/local.
```
rm -r /usr/local/*
    
```

Installing macOS via Software Update

Go to System Preferences > Software Update (macOS <= 12) or to System Settings > General > Software Update (macOS >= 13). Select to install the new macOS and follow the on-screen instructions.

Setup macOS

The steps are intent to be done (roughly) in order.

Check for system updates

Check for updates of macOS in System Preferences > Software Update (macOS <= 12) or System Settings > General > Software Update (macOS >= 13).

Xcode

A full Xcode installation is not always needed. Most often, the Xcode Command Line Tools (CLT) are enough, for example for Homebrew. But some development software needs a full Xcode installation such as the FreePGI Fortran Compiler. So one can, for example, install the Xcode CLT only, and install the full Xcode only if another program demands it. Note that installing the full XCode takes considerable time (count rather half an hour or more).

The normal way to install the XCode Command Line Tools (CLT) from the terminal would be:

xcode-select --install

There was a glitch (in macOS 13 Ventura) if you installed with Software Update and had a full XCode installation: the XCode Command Line Tools did not get updated so that gfortran, for example, clashes with XCode’s clang compiler. So I first uninstalled CLT and reinstalled them again:

sudo rm -rf /Library/Developer/CommandLineTools
sudo xcode-select --install

On macOS 14 Sonoma, this prints xcode-select: note: install requested for command line developer tools and you have to open Software Update again to install the command line tools.

This still pointed to the compiler within the XCode app:

xcode-select --print-path
# /Applications/Xcode.app/Contents/Developer/

This can be fixed by pointing to clang within the CLT:

sudo xcode-select -switch /Library/Developer/CommandLineTools

If you did a fresh install, you might let Homebrew do the job, i.e. it will install the XCode Command Line Tools if they are missing.

The full Xcode can be installed from the App Store. You have to open it once and confirm the Usage Agreement in order to use the bundled tools. If you install Xcode, it is reasonable to complete the Xcode installation and the one-time opening before starting with Homebrew. Otherwise, it might install the command line tools CLT twice, but it costs only download bandwidth and time.

While waiting for XCode to install, you can download and install the other apps from App Store, XQuartz and some Freeware except LaTeXiT and SourceTree. You can also Set preferences of macOS and standard apps.

Set preferences of macOS and standard apps

Set System Preferences (macOS <= 12) or System Settings (macOS >= 13) such as: Check for updates of macOS in System Preferences > Software Update (macOS <= 12) or System Settings > General > Software Update (macOS >= 13) .

Set computer name in Sharing > Computer Name or in General > Sharing > Local hostname
Unset all in Mission Control or in Desktop & Dock > Mission Control
Set Keyboard > Modifier Keys… > Caps Lock Key to No Action or set Keyboard > Keyboard Shortcuts > Modifier Keys > Caps Lock Key to No Action

Set preferences/settings in standard macOS apps such as:

Terminal
- Set Profiles > Shell > When the shell exists: to Close if the shell exited cleanly
- Unset tickbox Profiles > Advanced > Set locale environment variables on startup
Finder
- Set tickbox Advanced > Show all filename extensions

Set the default shell

Apple is now using zsh as its default shell. If you want to stay with bash, change it in the terminal:

chsh -s /bin/bash

To get rid of the nagging reminder that the default shell is now zsh every time you open a new terminal window, set in your .bash_profile:

export BASH_SILENCE_DEPRECATION_WARNING=1

My current .bash_profile is as dot-bash_profile in this repository along with the two sub-config files .bashrc as dot-bashrc for general aliases and functions and .bashrc.13 as dot-bashrc.13 for macOS 13 Ventura-specific aliases and functions.

Restore backup

If you did a fresh install, transfer files from your backup media either directly or using a Time Machine backup.

App Store

Even when you installed using Software Update, you should check for updates. Do not look only in App Store… > Updates but also on your account (on the bottom left) if there is an update. Xcode did not show up in Updates for me but I had to update it from the account page.

If you did a fresh install, check your account in the App Store. There are bundled apps such as Keynote, Numbers, etc., which you should open once to get over the welcome screens and offered tutorials.

It also shows you a list of previously installed apps on other systems. Install what you still want and open it once.

My essential apps from the App Store are:
Keynote/Pages/Numbers, The Unarchiver, WordService, HiddenMe, Microsoft Remote Desktop, Engauge Digitizer.

XQuartz

XQuartz is the X-window system running on macOS, needed for ∗nix GUI programs. Get it from XQuartz.

Homebrew

Install Homebrew for easy ∗nix package installation.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This installs Homebrew into /usr/local on macOS Intel and into /opt/homebrew on macOS Apple Silicon (M1, M2, etc.).

You might want to put into your .bash_profile for macOS Intel:

[[ -x $(which brew) ]] && eval $(brew shellenv)

and for macOS Apple Silicon (Mx):

[[ -d /opt/homebrew ]] && eval $(/opt/homebrew/bin/brew shellenv)

so that Homebrew can be found. This sets, for example, the environment variables HOMEBREW_PREFIX, HOMEBREW_CELLAR, and HOMEBREW_REPOSITORY and prepends the $PATH with Homebrew’s bin directory. I set this manually because I want to have Homebrew’s bin directory at the end rather than at the beginning of the system $PATH such as:

if [[ -d /opt/homebrew ]] ; then
    # eval $(/opt/homebrew/bin/brew shellenv)
    # or by hand to append rather than prepand path
    export HOMEBREW_PREFIX="/opt/homebrew";
    export HOMEBREW_CELLAR="/opt/homebrew/Cellar";
    export HOMEBREW_REPOSITORY="/opt/homebrew";
    export PATH=${PATH}:/opt/homebrew/bin
elif [[ -e /usr/local/bin/brew ]] ; then
    export HOMEBREW_PREFIX="/usr/local";
    export HOMEBREW_CELLAR="/usr/local/Cellar";
    export HOMEBREW_REPOSITORY="/usr/local";
    export PATH=${PATH}:/usr/local/bin
fi

Set your default shell #2

Apple moved to zsh because of the license change of bash from GPLv2 to GPLv3 with its version 4.0. The current bash shell on macOS is hence 3.2 from 2007. If you want to use the latest version of bash, install it with Homebrew, “whitelist” the new shell as a login shell, and choose it as your default login shell:

brew install bash

# add the following line to /etc/shells
# /usr/local/bin/bash
# or
# /opt/homebrew/bin/bash
sudo nano /etc/shells

chsh -s ${HOMEBREW_PREFIX}/bin/bash

Note that your shell scripts will probably still use the Apple default bash shell because they often have the shebang line #!/bin/bash. The most portable way to write shell scripts is to use #!/usr/bin/env bash as your shebang. This will take the first bash in your $PATH, which would now be /usr/local/bin/bash or /opt/homebrew/bin/bash.

You can now use bash-completion with the new bash shell.

brew install bash-completion@2

You then have to put the following lines in your .bash_profile to use bash-completion:

if [[ -f "${HOMEBREW_PREFIX}/etc/profile.d/bash_completion.sh" ]] ; then
    export BASH_COMPLETION_COMPAT_DIR="${HOMEBREW_PREFIX}/etc/bash_completion.d"
    source "${HOMEBREW_PREFIX}/etc/profile.d/bash_completion.sh"
fi

Note that these lines have to be after the sourcing of .fzf.bash in your .bash_profile if you installed fzf, otherwise you get an error such as programmable_completion: source: possible retry loop.

You can do the exact same steps for the zsh shell. Apples version of zsh is rather new but if you want to have the newest developments, install zsh with Homebrew, whitelist it and use it as your default shell. If you use zsh, you might want to check out Oh My ZSH for easy configuration of zsh.

After a system update such as from System Preferences > Software Update, there might be a link Relocated Items/ on your Desktop pointing to /Users/Shared/Relocated Items. This is a copy of the changed /etc/shells. As long as Apple does not modify /etc/shells during an update, the edited version stays untouched, though. One can safely delete the link on the Desktop and also the directory under /Users/Shared. It does not hurt to do a cat /etc/shells in the terminal before, checking that your edits are still there.

Homebrew #2

GNU compiler and netCDF software
The gcc suite includes gfortran. cmake is the build system of various software packages. Install nco, ncview, and Panoply to work with and visualise netCDF files. This installs the netcdf-C version, which comes with ncdump, etc. Homebrew had netcdf-C, netcdf-C++, and netcdf-Fortran bundled in netcdf before. They are individual packages now. netcdf installs the netcdf-C package only. See install_netcdf below for Fortran support.
```
brew install gcc
    
```
```
brew install cmake
    
```
```
for i in geos gdal nco ncview ; do brew install ${i} ; done
    
```
```
brew install cdo
    
```
```
brew install --cask temurin  # Java, for panoply
    
```
```
brew install --cask panoply
    
```
HDF5 from Homebrew is not thread-safe so cdo will need the -L flag if piping, i.e. more than one operator is given to cdo in one call such as cdo -timmean -selvar,Tair infile outfile. I use in my .bashrc:
```
alias cdo="cdo -L"
    
```
Note that cdo -L instead of purely cdo must also be used in scripts for piping.

Install more practical software

Some more practical software such as, fd for a faster find, ripgrep for grepping across a directory tree, the statistical computing environment R, the version control system subversion, and the command-line fuzzy finder fzf, and the tldr implementation tealdeer for more concise help pages:

brew install htop        # dynamic real-time information of running processes

brew install tree        # visualise folder tree structure

brew install fd          # faster find

brew install bat         # cat with syntax highlighting

brew install ripgrep     # grep across directory tree

brew install fzf         # command-line fuzzy finder

brew install wget        # retrieve files from web servers

brew install tealdeer    # simples help pages for command-line tools

brew install ghostscript # postscript and pdf language interpreter

brew install enscript    # convert text files to postscript files

brew install imagemagick # image manipulations

brew install ffmpeg      # for movies

brew install pandoc      # convert between markup languages

brew install pkg-config  # reveal details of installed libraries

brew install graphviz doxygen # documentation for programming languages

brew install subversion  # version control system

brew install git         # version control system

brew install r           # statistical computing environment

And some installations of the helpers:

${HOMEBREW_PREFIX}/opt/fzf/install

tldr --update

I also set ${HOME$/.tealdeer as my configuration directory for tealdeer in .bash_profile,

export TEALDEER_CONFIG_DIR=${HOME}/.tealdeer

and get a standard config file ${HOME}/.tealdeer/config.toml that I edit to tailor to my taste:

tldr --seed-config

Emacs

I used to use Aquamacs, then used Spacemacs, and then tried a few other setups (doom, dotemacs, boremacs, minemacs, etc.). Now I am using my own setup by copy/paste different bits from the other setups. My current setup is in dot-emacs.d in this repository.

I install Emacs with Homebrew:

brew install --cask emacs

Coming from another Emacs, backup .emacs and .emacs.d:

cd ${HOME}
if [[ -f .emacs ]] ; then mv .emacs .emacs.bak ; fi
if [[ -d .emacs.d ]] ; then mv .emacs.d .emacs.d.bak ; fi

Then I install my setup by copying it to ~/.emacs.d.

I use the font Source Code Pro that is also used in Spacemacs and install aspell for spell checking:

brew tap homebrew/cask-fonts
brew install --cask homebrew/cask-fonts/font-source-code-pro

brew install aspell

If you open the new Emacs for the first time, it will install and byte-compile some packages. This might take some time.

LaTeX

One can download LaTeX from MacTeX or use a Homebrew cask. I have chosen Homebrew’s cask this time because I use the BasicTeX installation and I let Homebrew handle the update between years (brew upgrade --cask basictex), which is always a hassle otherwise:

brew install --cask basictex

If you chose BasicTeX, then I install a few LaTeX packages, which I encountered during different projects:

sudo tlmgr update --self ; \
sudo tlmgr install \
    wasysym german titlesec wasy elsarticle \
    supertabular lineno helvetic textpos multirow subfigure appendix \
    lipsum dinbrief a0poster wallpaper collection-fontsrecommended \
    dvipng kastrup boondox newtx type1cm ucs dvipng a0poster floatflt \
    enumitem lastpage hyphenat footmisc simplekv chemfig units \
    ntheorem algorithms cleveref a4wide lettrine mdframed \
    needspace preprint xifthen ifmtarg algorithmicx changepage \
    sidecap sttools marginnote draftwatermark everypage fontinst \
    fltpoint tabfigures mnsymbol mdsymbol collection-fontutils \
    fontaxes was pdfcrop latexmk fncychap tabulary varwidth \
    framed capt-of makecell xstring moreverb wrapfig \
    adjustbox collectbox threeparttable capt-of pgf simplekv \
    cmbright tcolorbox environ titling gensymb program breakurl \
    ncctools vruler apacite biblatex biber blindtext pgfgantt \
    biblatex-chicago biblatex-apa fontawesome5 cormorantgaramond \
    sectsty pdfcol soul anyfontsize doublestroke doublestroke \
    arydshln

I also install LaTeXML so that Emacs’ org-mode can convert LaTeX equations to MathML on export:

brew install latexml

Freeware

Some essential Freeware for me:

AppCleaner, for removing apps and all their traces,
Adobe Reader, because Preview has problems with some PDFs,
Firefox Developer Edition, Safari is not always supported. Chrome is probably the most supported browser. I sometimes also use Opera,
LaTeXiT, exporting LaTeX equations as graphics,
Rectangle, moving windows with keystrokes,
Zotero, reference manager,
Skype, video calls,
SourceTree, git GUI originally for bitbucket but works with other git repositories as well,
Spotify, streaming music,
VLC, video player for all formats,
Engauge Digitizer, recover data points from graphs.

Myriad Pro

I like the Myriad Pro font and AGU journals currently use it. The Myriad Pro font comes with the Adobe Acrobat Reader.

To install for non-LaTeX programs, one can install in Font Book the four otf-files from the directory ‘/Applications/Adobe Acrobat Reader DC.app/Contents/Resources/Resource/Font’.

An extended set of glyphs are given in the zip file ‘MyriadPro.zip’:
unzip MyriadPro.zip and drag the folder with the .otf files into Font Book.

To install Myriad Pro for LaTeX, using the Adobe fonts, one can launch the following commands in terminal:

for i in fontinst fltpoint tabfigures mnsymbol mdsymbol \
    collection-fontutils ; do \
    sudo tlmgr install ${i} ; done
git clone https://github.com/sebschub/FontPro.git
cd FontPro
mkdir otf
FONT=MyriadPro
cp "/Applications/Adobe Acrobat Reader.app/Contents/Resources/Resource/Font/"${FONT}*.otf otf/
./scripts/makeall ${FONT}
echo y | sudo ./scripts/install
sudo updmap-sys --enable Map=${FONT}.map
sudo -H mktexlsr
kpsewhich ${FONT}.map
cd ..
\rm -fr FontPro

Python

macOS Catalina (10.15) still comes with Python version 2.7.16 as its default version. Official support for Python 2 has ended Januar 2020. So you want to install Python 3. From macOS Big Sur (11.5) onwards, macOS comes with Python 3 (from macOS 13 Ventura, is is actually part of the XCode command line tools). But I still recommend to install Python with pyenv and pyenv-virtualenv: you can install different Python versions, use very easily different virtual environments in different directories (projects), etc.

Installation of Python versions can be a real mess at times, as noted by XKCD:

pyenv and pyenv-virtualenv makes that very easy: see the great article pyenv: Multi-version Python development on Mac by Dirk Avery.

To install pyenv with Homebrew:

brew install openssl readline sqlite3 xz zlib
brew install pyenv
brew install pyenv-virtualenv

You have to set the following in your .bash_profile so that the shell always finds the currently chosen Python version as the first entry.

export PYENV_ROOT="${HOME}/.pyenv"
export PATH=${PYENV_ROOT}/shims:${PATH}
if command -v pyenv 1>/dev/null 2>&1 ; then eval "$(pyenv init -)" ; fi
if command -v pyenv virtualenv-init 1>/dev/null 2>&1 ; then eval "$(pyenv virtualenv-init -)" ; fi

Note that the $PATH environment must be prepended with ${PYENV_ROOT}/shims and not ${PYENV_ROOT}/bin as given in the (older) user guide. Also, this has to be done after the addition of Homebrew to the system $PATH so that the pyenv Python installation can be found first; otherwise Homebrew’s Python will be found first.

pyenv and pyenv-virtualenv 101
After starting a new shell, for example by doing exec ${SHELL}, you can start installing and using different Python versions (but see my installation below):
```
pyenv install --list
pyenv install 3.12.1
pyenv rehash
pyenv global 3.12.1
    
```
Remember that you always have to rehash after you installed a new version or new virtual environment.

Try to rehash first if a problem occurs with pyenv. For example, some new Homebrew packages might upgrade pyenv as well. Then you get an error such as
```
/Users/cuntz/.pyenv/shims/python: line 21:
/usr/local/Cellar/pyenv/2.3.12/libexec/pyenv:
No such file or directory
    
```
pyenv rehash resolves the issue.

Note that I actually install Python versions currently as follows:
```
brew install tcl-tk
    
```
```
env PYTHON_CONFIGURE_OPTS=" \
    --with-tcltk-includes='-I${HOMEBREW_PREFIX}/opt/tcl-tk/include' \
    --with-tcltk-libs='-L${HOMEBREW_PREFIX}/opt/tcl-tk/lib -ltcl8.6 -ltk8.6' \
    --enable-optimizations --enable-framework=${HOME}/Library/Frameworks" \
    CFLAGS="-I$(brew --prefix xz)/include" \
    LDFLAGS="-L$(brew --prefix xz)/lib" \
    PKG_CONFIG_PATH="$(brew --prefix xz)/lib/pkgconfig" \
    pyenv install 3.12.1
    
```
```
pyenv rehash
    
```
This uses the newer Tcl/Tk version from Homebrew for tkinter, turns on profile guided optimization and link time optimization for Python being about 10% faster, and enables a framework build. Optimization takes significantly more time to install Python. If you do not use tkinter and just want to try out a Python version, pyenv install 3.12.1 is just fine. Otherwise I recommend the optimization because you do not install Python too often but get a significant gain. Python has to be a framework (--enable-framework) if wxPython is used (see the pyenv wiki for details). This needs administrator rights since macOS 13 Ventura. Each installation of packages, creation of virtual environments, etc. will need sudo then. This is a nuisance. So I rather install the framework in my local Library folder (--enable-framework=${HOME}/Library/Frameworks). This only works with pyenv version >= 2.3.12, otherwise it fails due to a bug in pyenv.

Virtual environments are created as:
```
pyenv virtualenv 3.12.1 pytest
pyenv rehash
    
```
The virtual environment pytest can then be used just as any installed Python version with pyenv. For example:
```
pyenv local pytest
    
```
pyenv provides also anaconda and miniconda, with which you can use conda environments:
```
pyenv install miniconda3-4.7.12
pyenv rehash
pyenv global miniconda3-4.7.12
pyenv virtualenv pytest
    
```
You can then install (conda and pip) packages in the test project, etc.:
```
conda install numpy scipy matplotlib
    
```
If git tells gettext not found after installing anaconda/miniconda, see Nicolas Hery.

Essential Python packages

Anaconda comes with hundreds of packages. I tend to use either an official Python version or miniconda and install my essential packages with pip or conda/mamba. These are currently in my main environment:
numpy, scipy, matplotlib, cartopy, ipython, jupyter, pandas, cftime, netcdf4, udunits, statsmodels, scikit-learn, xlrd, openpyxl, mpi4py, schwimmbad, xarray, numpydoc, pytest, pytest-cov, flake8, gdal, f90nml, pykdtree, cython, pyshp, six, wheel

and mostly this subset in other virtual environments:
numpy, scipy, matplotlib, pandas, netcdf4, xarray, ipython, flake8, wheel

You would have to source your .bash_profile again if you install with pip or conda just after installing pyenv.

There is currently a problem with Apple’s Accelerate framework on Apple Silicon (M1/2) so that one should use OpenBLAS. You also need to tell pip where to find the HDF5 library and the GEOS library and the like. You do not need the lines with OPENBLAS and HDF5-DIR if you are on macOS on Intel and Homebrew installs into /usr/local; this directory is searched automatically.

   # essential subset
   if [[ "$(uname -m)" == "arm64" ]] ; then
	   export OPENBLAS="$(brew --prefix openblas)"
	   export HDF5_DIR="$(brew --prefix hdf5)"
	   export GEOS_DIR="$(brew --prefix geos)"
	   export GEOS_CONFIG="$(brew --prefix geos)/bin/geos-config"
   fi
   pyenv virtualenv 3.12.1 pystd
   pyenv rehash
   pyenv global pystd
   pyenv rehash
   # test if install works
   python -m pip install numpy
   for i in wheel scipy matplotlib ipython pandas netcdf4 \
	   xarray ipython flake8 ; do \
	   python -m pip install ${i} ; done
   # other standard packages
   for i in jupyter statsmodels scikit-learn schwimmbad \
	   numpydoc pytest pytest-cov f90nml pykdtree cython pyshp six \
	   xlrd openpyxl ; do \
	   python -m pip install ${i} ; done
   # shapely and gdal for cartopy
   # shapely needs to be built from source to link to geos.
   # Uninstall it if already installed
   [[ -z $(python -m pip freeze | grep shapely) ]] && \
	   python -m pip uninstall -y shapely
   python -m pip install shapely --no-binary shapely
   # gdal needs to know the installed gdal version
   python -m pip install GDAL==$(gdal-config --version) \
	   --global-option=build_ext --global-option="-I${HOMEBREW_PREFIX}/include"
   python -m pip install cartopy
   # basemap is back in development again :-)
   # but does not work with Python 3.12 yet :-(
   # python -m pip install basemap-data
   # python -m pip install basemap-data-hires
   # python -m pip install basemap
   # if you use GUIs written in wxpython
   CXXFLAGS="-I${HOMEBREW_PREFIX}/include" python -m pip install wxpython
   # mpi4py will be installed after installing openmpi later

Replace python -m pip with conda, or better mamba after installing it, if using conda environments instead of /pip.

locate

Create locate database so that you can search files with the locate command:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.locate.plist

This might already be running (Operation already in progress or Load failed: 5: Input//output error) if you upgraded via System Update.

Payware

Install Payware, which is for me:

Microsoft Office,
Antidote, spell and grammar checker for English and French,
NAG compiler, very meticulous Fortran compiler,
Intel oneAPI Base Toolkit and Intel oneAPI HPC Toolkit, C/C++/Fortran compiler producing very fast code,
CyberGhost, VPN client,
IDL, interactive data language.

Additional software

Install additional software from you institution or similar such as VPN clients, cloud services, etc. For INRAE this is: WithSecure Antivirus, GlobalProtect VPN, StorageMadeEasy.

netcdf4_fortran and openmpi development packages

You can install netcdf-fortran for the gfortran compiler.

brew install netcdf-fortran

This will automatically update netcdf-fortran for gfortran if a newer version of netcdf-C and/or netcdf-fortran becomes available.

However, if you use other Fortran compilers as well, you might want to use the script install_netcdf to install it and not flood your namespace with different versions of netcdf.mod, etc. The script install_netcdf installs netcdf-fortran, openmpi, and/or mpich development packages for different Fortran compilers. The script is well documented and we just describe the general steps.

Look for the latest versions (numbers) of netcdf-fortran, openmpi, and/or mpich (addresses are also given at the beginning of the script install_netcdf) and set them below donetcdf4_fortran, doopenmpi, and/or dompich.
Set donetcdf4_fortran, doopenmpi, and/or dompich to 1.
Check that prefix=/usr/local.
Set Fortran compiler, e.g. ~fortran_compilers=”gfortran”~.
For Intel, you need to source the compiler setup script such as:

source /opt/intel/bin/compilervars.sh intel64

For PGI, you also have to set the pgipath.
Run he script on the command line and give your sudo password if you install into prefix=/usr/local.

After having installed openmpi, one can also install mpi4py in Python, for example:

env MPICC=/usr/local/openmpi-4.1.6-gfortran/bin/mpicc python -m pip install mpi4py

However, homebrew upgrades also netcdf-C to newer versions if you install or update a package that depends on it. Then the netcdf-fortran package installed with install_netcdf will not work anymore (it will link to the old, uninstalled version) and you have to rerun the script install_netcdf with donetcdf4_fortran=1 and ~fortran_compilers=”gfortran”~. I still do it this way to minimize conflicts between different Fortran compilers and re-installing netcdf-fortran with install_netcdf for gfortran is very fast.

mcuntz / setup_mac

Setting up macOS for scientific computing and software development

Table of Contents

How to update the OS

Fresh install

Before starting

Make a bootable installer for macOS on USB

Installing macOS with the bootable installer

Update via Software Update

Before starting

Installing macOS via Software Update

Setup macOS

Check for system updates

Xcode

Set preferences of macOS and standard apps

Set the default shell

Restore backup

App Store

XQuartz

Homebrew

Set your default shell #2

Homebrew #2

Emacs

LaTeX

Freeware

Myriad Pro

Python

locate

Payware

Additional software

netcdf4_fortran and openmpi development packages

About

Languages