Installation problem on Python 3.10 on Apple Silicon Mac

[Kenny Song]

Copying a message I received outside of GitHub. It looks like we haven't fully solved the Rust compiler issue.

---

On

> python --version
Python 3.10.11

I get with

> pip freeze | grep token
tokenizers==0.15.0

and yet, when I run:

> pip install langcheck

it still tries to install tokenizers, from source.
I guess it comes from:

Collecting tokenizers!=0.11.3,<0.13,>=0.11.1 (from transformers>=4.6->langcheck)1

since I have transformers==4.35.2 in that environment.

So I end up with

error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-310-darwin.so'` failed with code 101
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

The confusing part here is that if I run

> pip install -U transformers

I still stay with

Requirement already satisfied: transformers in /opt/homebrew/Caskroom/miniforge/base/envs/truesilicon/lib/python3.10/site-packages (4.35.2)

that is ACTUALLY the current version: https://github.com/huggingface/transformers/tags

Where is 'transformers >= 4.6' in the dependencies coming from?