Here are some of the fixes that helped me install popular ML/RL libraries on apple silicon. I'm sharing it here in hopes that it's useful for others as well.
here's a way to avoid interacting with conda mini-forge: https://stackoverflow.com/questions/65383338/zsh-illegal-hardware-instruction-python-when-installing-tensorflow-on-macbook/68214296#68214296
should just work with pip3 install torch.
See: pytorch/pytorch#56333
usually the issue is I can't build scipy, here's a fix for that:
scipy/scipy#13409
have to change CMakeLists.txt and compile from source, also change the C Compiler to Clang: openai/procgen#69
Modify CMkaeLists.txt with the options here: https://stackoverflow.com/questions/7031126/switching-between-gcc-and-clang-llvm-using-cmake