Initializing models... Error: unknown error

Question

Initializing models... Error: unknown error

milsun opened this issue 4 months ago · comments

Milan commented 4 months ago

When using Swift CLI example, followed the exact commands mentioned, getting above error:

Chip: Apple M3 Max
OS: 14.3.1
Xcode: Version 15.2
Apple Swift version: 5.9.2

Atila Orhon · Answer 1 · Thu Feb 29 2024 15:57:01 GMT+0800 (China Standard Time)

Hi @milsun Can you share the full logs from the terminal?

Milan · Answer 2 · Thu Feb 29 2024 16:09:51 GMT+0800 (China Standard Time)

(base) volleedemo@Vollees-MacBook-Pro vollee-demo % git clone https://github.com/argmaxinc/whisperkit.git
cd whisperkit
Cloning into 'whisperkit'...
remote: Enumerating objects: 367, done.
remote: Counting objects: 100% (189/189), done.
remote: Compressing objects: 100% (104/104), done.
remote: Total 367 (delta 88), reused 115 (delta 55), pack-reused 178
Receiving objects: 100% (367/367), 901.56 KiB | 6.39 MiB/s, done.
Resolving deltas: 100% (135/135), done.
(base) volleedemo@Vollees-MacBook-Pro whisperkit % make setup
make download-model MODEL=large-v3
Setting up environment...
/Users/volleedemo/miniconda3/bin/pip3
/Users/volleedemo/miniconda3/bin/python3
Requirement already satisfied: huggingface_hub in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (0.21.2)
Requirement already satisfied: filelock in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from huggingface_hub) (3.13.1)
Requirement already satisfied: fsspec>=2023.5.0 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from huggingface_hub) (2024.2.0)
Requirement already satisfied: requests in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from huggingface_hub) (2.31.0)
Requirement already satisfied: tqdm>=4.42.1 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from huggingface_hub) (4.65.0)
Requirement already satisfied: pyyaml>=5.1 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from huggingface_hub) (6.0.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from huggingface_hub) (4.10.0)
Requirement already satisfied: packaging>=20.9 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from huggingface_hub) (23.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from requests->huggingface_hub) (2.0.4)
Requirement already satisfied: idna<4,>=2.5 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from requests->huggingface_hub) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from requests->huggingface_hub) (1.26.18)
Requirement already satisfied: certifi>=2017.4.17 in /Users/volleedemo/miniconda3/lib/python3.11/site-packages (from requests->huggingface_hub) (2023.11.17)
Checking for Homebrew...
Homebrew is installed.
Checking for git-lfs...
git-lfs is installed.
Done 🚀
Downloading model large-v3...
Setting up repository...
Repository not found, initializing...
Cloning into './Models/whisperkit-coreml'...
remote: Enumerating objects: 483, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 483 (delta 0), reused 0 (delta 0), pack-reused 466
Receiving objects: 100% (483/483), 1.25 MiB | 1.47 MiB/s, done.
Resolving deltas: 100% (138/138), done.
Fetching model large-v3...
(base) volleedemo@Vollees-MacBook-Pro whisperkit %
(base) volleedemo@Vollees-MacBook-Pro whisperkit %
(base) volleedemo@Vollees-MacBook-Pro whisperkit %
(base) volleedemo@Vollees-MacBook-Pro whisperkit % swift run transcribe --model-path "Models/whisperkit-coreml/openai_whisper-large-v3" --stream
Fetching https://github.com/apple/swift-argument-parser.git from cache
Fetching https://github.com/huggingface/swift-transformers.git from cache
Fetched https://github.com/apple/swift-argument-parser.git (1.27s)
Fetched https://github.com/huggingface/swift-transformers.git (1.28s)
Computing version for https://github.com/apple/swift-argument-parser.git
Computed https://github.com/apple/swift-argument-parser.git at 1.3.0 (0.23s)
Computing version for https://github.com/huggingface/swift-transformers.git
Computed https://github.com/huggingface/swift-transformers.git at 0.1.2 (0.26s)
Computed https://github.com/apple/swift-argument-parser.git at 1.3.0 (0.00s)
Computed https://github.com/huggingface/swift-transformers.git at 0.1.2 (0.00s)
Creating working copy for https://github.com/apple/swift-argument-parser.git
Working copy of https://github.com/apple/swift-argument-parser.git resolved at 1.3.0
Creating working copy for https://github.com/huggingface/swift-transformers.git
Working copy of https://github.com/huggingface/swift-transformers.git resolved at 0.1.2
Building for debugging...
/Users/volleedemo/Desktop/vollee-demo/whisperkit/Sources/WhisperKit/Core/TokenSampler.swift:47:29: warning: conformance of 'Float16' to 'BNNSScalar' is unavailable in macOS; this is an error in Swift 6
scalarType: FloatType.self, // FIXME: Float16 here breaks in swift 6
^
Accelerate.Float16:4:11: note: conformance of 'Float16' to 'BNNSScalar' has been explicitly marked unavailable here
extension Float16 : BNNSScalar {
^
/Users/volleedemo/Desktop/vollee-demo/whisperkit/Sources/WhisperKit/Core/TokenSampler.swift:56:33: warning: conformance of 'Float16' to 'BNNSScalar' is unavailable in macOS; this is an error in Swift 6
scalarType: FloatType.self,
^
Accelerate.Float16:4:11: note: conformance of 'Float16' to 'BNNSScalar' has been explicitly marked unavailable here
extension Float16 : BNNSScalar {
^
[96/96] Linking transcribe
Build complete! (8.63s)
Initializing models...
Error: unknown error

Milan · Answer 3 · Thu Feb 29 2024 17:26:14 GMT+0800 (China Standard Time)

Tried with all available tags too, same error.

Atila Orhon · Answer 4 · Fri Mar 01 2024 01:44:12 GMT+0800 (China Standard Time)

Thanks @milsun, could you also share your HW details? Is this an Apple Silicon Mac?

Zach Nagengast · Answer 5 · Fri Mar 01 2024 02:15:00 GMT+0800 (China Standard Time)

@milsun One more thing that would help us would be adding the --verbose flag:

swift run transcribe --model-path "Models/whisperkit-coreml/openai_whisper-large-v3" --stream --verbose

And paste the output of that here.

Milan · Answer 6 · Fri Mar 01 2024 13:40:24 GMT+0800 (China Standard Time)

swift run transcribe --model-path "Models/whisperkit-coreml/openai_whisper-large-v3" --stream --verbose

Fetching https://github.com/huggingface/swift-transformers.git from cache
Fetching https://github.com/apple/swift-argument-parser.git from cache
Fetched https://github.com/apple/swift-argument-parser.git (1.19s)
Fetched https://github.com/huggingface/swift-transformers.git (1.19s)
Computing version for https://github.com/apple/swift-argument-parser.git
Computed https://github.com/apple/swift-argument-parser.git at 1.3.0 (0.03s)
Computing version for https://github.com/huggingface/swift-transformers.git
Computed https://github.com/huggingface/swift-transformers.git at 0.1.2 (0.02s)
Computed https://github.com/apple/swift-argument-parser.git at 1.3.0 (0.00s)
Computed https://github.com/huggingface/swift-transformers.git at 0.1.2 (0.00s)
Creating working copy for https://github.com/apple/swift-argument-parser.git
Working copy of https://github.com/apple/swift-argument-parser.git resolved at 1.3.0
Creating working copy for https://github.com/huggingface/swift-transformers.git
Working copy of https://github.com/huggingface/swift-transformers.git resolved at 0.1.2
Building for debugging...
/Users/volleedemo/Desktop/vollee-demo/whisperkit/Sources/WhisperKit/Core/TokenSampler.swift:47:29: warning: conformance of 'Float16' to 'BNNSScalar' is unavailable in macOS; this is an error in Swift 6
scalarType: FloatType.self, // FIXME: Float16 here breaks in swift 6
^
Accelerate.Float16:4:11: note: conformance of 'Float16' to 'BNNSScalar' has been explicitly marked unavailable here
extension Float16 : BNNSScalar {
^
/Users/volleedemo/Desktop/vollee-demo/whisperkit/Sources/WhisperKit/Core/TokenSampler.swift:56:33: warning: conformance of 'Float16' to 'BNNSScalar' is unavailable in macOS; this is an error in Swift 6
scalarType: FloatType.self,
^
Accelerate.Float16:4:11: note: conformance of 'Float16' to 'BNNSScalar' has been explicitly marked unavailable here
extension Float16 : BNNSScalar {
^
[96/96] Linking transcribe
Build complete! (8.42s)
Initializing models...
[WhisperKit] Running on arm64
[WhisperKit] Loading models...
[WhisperKit] Loading models from /Users/volleedemo/Desktop/vollee-demo/whisperkit/Models/whisperkit-coreml/openai_whisper-large-v3 with prewarmMode: false
[WhisperKit] Loading feature extractor
[WhisperKit] Loaded feature extractor
[WhisperKit] Loading audio encoder
[WhisperKit] Loaded audio encoder
[WhisperKit] Loading text decoder
[WhisperKit] Loaded text decoder
[WhisperKit] Loading tokenizer for large-v3
Error: unknown error

Milan · Answer 7 · Fri Mar 01 2024 13:49:01 GMT+0800 (China Standard Time)

Thanks @milsun, could you also share your HW details? Is this an Apple Silicon Mac?

My first comment has all the HW details.

Milan · Answer 8 · Sat Mar 02 2024 14:19:34 GMT+0800 (China Standard Time)

Can you share the specific versions, for which it works, need to get it in place for a project I have been working on. Thanks!

Atila Orhon · Answer 9 · Sat Mar 02 2024 14:46:53 GMT+0800 (China Standard Time)

Thanks for adding the HW info in your earlier comment. M1 and newer (all Apple Silicon Macs) should definitely work for large-v3.

Atila Orhon · Answer 10 · Sat Mar 02 2024 14:50:34 GMT+0800 (China Standard Time)

@milsun I tested this command on M3 Max + macOS 14.4 which is the closest configuration I have to yours and it worked for me.

We think that you might have been impacted by an issue that tries to pull the tokenizer assets from HuggingFace Hub and they had an outage around the same time you filed this issue. Could you please try again and report back whether the problem persists?

Milan · Answer 11 · Sat Mar 02 2024 20:11:14 GMT+0800 (China Standard Time)

Did a fresh clone and install, still the same issue.

Atila Orhon · Answer 12 · Sun Mar 03 2024 09:09:30 GMT+0800 (China Standard Time)

Since I am unable to reproduce even when using similar hardware and OS versions, let's try a few different commands on your side to localize the issue. Could you try changing the model version from large-v3 to large-v2 and tiny respectively and report back?

Naga Sai Abhinay · Answer 13 · Mon Mar 11 2024 01:08:03 GMT+0800 (China Standard Time)

@atiorh I have the same issue as milsun.

Hardware:

M1 Macbook Pro 8GB
macOS 14.1.1

Setup:

1. Cloned the repo
2. Ran make to setup project and download the models

Logs

tiny

swift run transcribe --model-path "Models/whisperkit-coreml/openai_whisper-tiny" --audio-path ./Tests/WhisperKitTests/Resources/es_test_clip.wav --verbose

Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build complete! (0.26s)
Transcribing audio at Tests/WhisperKitTests/Resources/es_test_clip.wav -- file:///Users/nagasaiabhinaydevarinti/Documents/WhisperKit/
Initializing models...
[WhisperKit] Running on arm64
[WhisperKit] Loading models...
[WhisperKit] Loading models from /Users/nagasaiabhinaydevarinti/Documents/WhisperKit/Models/whisperkit-coreml/openai_whisper-tiny with prewarmMode: false
[WhisperKit] Loading feature extractor
[WhisperKit] Loaded feature extractor
[WhisperKit] Loading audio encoder
[WhisperKit] Loaded audio encoder
[WhisperKit] Loading text decoder
[WhisperKit] Loaded text decoder
[WhisperKit] Loading tokenizer for tiny
Error: unknown error

large-v3

swift run transcribe --model-path "Models/whisperkit-coreml/openai_whisper-large-v3" --audio-path ./Tests/WhisperKitTests/Resources/es_test_clip.wav --verbose

Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build complete! (0.15s)
Transcribing audio at Tests/WhisperKitTests/Resources/es_test_clip.wav -- file:///Users/nagasaiabhinaydevarinti/Documents/WhisperKit/
Initializing models...
[WhisperKit] Running on arm64
[WhisperKit] Loading models...
[WhisperKit] Loading models from /Users/nagasaiabhinaydevarinti/Documents/WhisperKit/Models/whisperkit-coreml/openai_whisper-large-v3 with prewarmMode: false
[WhisperKit] Loading feature extractor
[WhisperKit] Loaded feature extractor
[WhisperKit] Loading audio encoder
[WhisperKit] Loaded audio encoder
[WhisperKit] Loading text decoder
[WhisperKit] Loaded text decoder
[WhisperKit] Loading tokenizer for large-v3
Error: unknown error

large-v2

swift run transcribe --model-path "Models/whisperkit-coreml/openai_whisper-large-v2" --audio-path ./Tests/WhisperKitTests/Resources/es_test_clip.wav --verbose

Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build complete! (0.24s)
Transcribing audio at Tests/WhisperKitTests/Resources/es_test_clip.wav -- file:///Users/nagasaiabhinaydevarinti/Documents/WhisperKit/
Initializing models...
[WhisperKit] Running on arm64
[WhisperKit] Loading models...
[WhisperKit] Loading models from /Users/nagasaiabhinaydevarinti/Documents/WhisperKit/Models/whisperkit-coreml/openai_whisper-large-v2 with prewarmMode: false
[WhisperKit] Loading feature extractor
[WhisperKit] Loaded feature extractor
[WhisperKit] Loading audio encoder
[WhisperKit] Loaded audio encoder
[WhisperKit] Loading text decoder
[WhisperKit] Loaded text decoder
[WhisperKit] Loading tokenizer for large-v2
Error: unknown error

Also FYI, I ran make test --filter UnitTests to see if tokenizer loading would fail but all tests ran successfully.

Naga Sai Abhinay · Answer 14 · Mon Mar 11 2024 01:20:46 GMT+0800 (China Standard Time)

Update:

After running swift test --filter UnitTests, transcribe with tiny model: swift run transcribe --model-path "Models/whisperkit-coreml/openai_whisper-tiny" --audio-path ./Tests/WhisperKitTests/Resources/es_test_clip.wav --verbose is working.
large-v2 fails with the same logs as above
large-v3 works too. But I ran swift test --filter FunctionalTests and killed it after sometime. I assume testInitLarge() was executed and only post that large-v3 worked.

I'll see if I can reproduce this with a fresh install.