.tflite files support
stefangrotz opened this issue · comments
After the mozilla layoffs, the deepspeech team forked the Deepspeech repo and founded the company Coqui AI (https://github.com/coqui-ai/STT) where they continue the development and AFAIK they now only allow .tflite files to export models. It theoretically should work with the old code, but for me it didn't.
When I try to run it like this:
python3 autosub/main.py --file /Users/sgrotz/Downloads/kp193-hejma-auxtomatigo.mp3 --split-duration 8
with a .tflite file in the main folder and NO language model.
Then I get:
AutoSub
['autosub/main.py', '--file', '/Users/sgrotz/Downloads/kp193-hejma-auxtomatigo.mp3', '--split-duration', '8']
ARGS: Namespace(dry_run=False, file='/Users/sgrotz/Downloads/kp193-hejma-auxtomatigo.mp3', format=['srt', 'vtt', 'txt'], model=None, scorer=None, split_duration=8.0)
Warning no models specified via --model and none found in local directory. Please run getmodel.sh convenience script from autosub repo to get some.
Error: Must have pbmm model. Exiting
Have I done anything wrong here or doesn't AutoSub support .rflite files?
I tested it on MacOS and installed ffmpeg via homebrew.
Update: adding --model output_graph.tflite
works in the beginning, but then this happens:
AutoSub
['autosub/main.py', '--file', '/Users/sgrotz/Downloads/kp193-hejma-auxtomatigo.mp3', '--model', 'output_graph.tflite']
ARGS: Namespace(dry_run=False, file='/Users/sgrotz/Downloads/kp193-hejma-auxtomatigo.mp3', format=['srt', 'vtt', 'txt'], model='output_graph.tflite', scorer=None, split_duration=5)
model: /Users/sgrotz/Downloads/AutoSub/output_graph.tflite
Warning no scorers specified via --scorer and none found in local directory. Please run getmodel.sh convenience script from autosub repo to get some.
scorer:Input file: /Users/sgrotz/Downloads/kp193-hejma-auxtomatigo.mp3
Creating file: /Users/sgrotz/Downloads/AutoSub/output/kp193-hejma-auxtomatigo.srt
Creating file: /Users/sgrotz/Downloads/AutoSub/output/kp193-hejma-auxtomatigo.vtt
Creating file: /Users/sgrotz/Downloads/AutoSub/output/kp193-hejma-auxtomatigo.txt
Extracted audio to audio/kp193-hejma-auxtomatigo.wav
Splitting on silent parts in audio fileRunning inference:
TensorFlow: v2.3.0-6-g23ad988fcd
DeepSpeech: v0.9.3-0-gf2e9c858
Warning: reading entire model file into memory. Transform model file into an mmapped graph to reduce heap usage.
2021-11-14 19:04:06.298928: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Data loss: Can't parse /Users/sgrotz/Downloads/AutoSub/output_graph.tflite as binary proto
Invalid model file. Exiting
Same behavior on Colab, so it is not a MacOS issue.
Hi
There is a different DeepSpeech package for tflite models. You can install it via:
$ pip install --user --update deepspeech-tflite
Thanks for the answer :) But it still doesn't work, this is how I installed everything on Colab:
%cd /content/AutoSub
!python3 -m venv sub
!source sub/bin/activate
!pip3 install -r requirements.txt
!pip3 install --user --update deepspeech-tflite
It works now, the trick was not to use a venv and to remove --user --update
:
%cd /content/AutoSub
#!python3 -m venv sub
#!source sub/bin/activate
!pip3 install -r requirements.txt
!pip install deepspeech-tflite
Thanks for your help with my beginner problem :)
Hi @stefangrotz, thanks for documenting your experiences! Here's an updated recipe for those wanting to use Coqui models with Docker:
- Download
.pbm
and.tflite
from coqui (e.g. https://coqui.ai/english/coqui/v0.9.3) - Add
RUN pip3 install --user --update deepspeech-tflite
toDockerfile
- Add
COPY *.tflite ./
toDockerfile
- Rebuild container, and add
--model model.tflite
when starting
One question for @stefangrotz @abhirooptalasila - the instructions above are just using the newer Coqui models with the existing DeepSpeech application, right? Would there be an advantage to using the STT toolkit instead of DeepSpeech? If so, any thoughts on what updating AutoSub to use it would look like?
Hi
I was planning on implementing either one of Wav2Vec or NeMo as an addition to DeepSpeech #44.
I'm not sure of the performance difference between Coqui, DeepSpeech, and the above two models. If you do, please let me know. The last time I tried Wav2Vec, the accuracy was much better than DeepSpeech.
Coqui is not hard to implement as the codebase is similar to DeepSpeech.
I do not have statistics, but I would assume Coqui is better than DeepSpeech and Coqui comes with a wide variety of language models. Coqui would be the simplest way to expand the functionality of AutoSub.
Edit: wav2vec-U, wav2vec 2.0, and NeMo look good too. It would be great if the AutoSub user could pick from any of these backends.
I would add Vosk to the list, it works very well and has srt creation script out of the box.
But to keep things simple, I would say switching to Coqui might be a good first step since it is actively supported by a company while Deepspeech is abandoned by Mozilla.
I gave Coqui STT a try.
sed -i 's/deepspeech/stt/g' autosub/utils.py
python3 autosub/main.py --model ~/coquistt/models/model.tflite --scorer ~/coquistt/models/huge-vocabulary.scorer --file ~/coquistt/example.webm
It seems to work as a drop-in replacement.
Splitting on silent parts in audio file
Running inference:
TensorFlow: v2.3.0-14-g4bdd3955115
Coqui STT: v1.0.0-0-g27584037
7%|███████▎ | 20/280 [01:43<12:41, 2.93s/it]
There is more to do of course to make the switch, but it looks like it works conceptually.
Edit: It completed and I was able to compare the transcript between default DeepSpeech 0.9.3 and Coqui STT 1.0.0. Coqui STT was more accurate with complex words and they were about the same with one syllable words. Overall worth upgrading.
Vosk includes an example python script to generate an srt file. I got that to work too.
pip3 install vosk
git clone https://github.com/alphacep/vosk-api
cd vosk-api/python/example
wget https://alphacephei.com/kaldi/models/vosk-model-small-en-us-0.15.zip
unzip vosk-model-small-en-us-0.15.zip
mv vosk-model-small-en-us-0.15 model
python3 test_srt.py test.webm > test.srt
Have some free time right now, so I will add Coqui support as a starter.
@stefangrotz @TechnologyClassroom
Can you check the changes I pushed?
Cleaned up some stuff.
@abhirooptalasila It mostly looks good to me.
autosub/utils.py
in 40bb833#diff-3a061d9e61ec5b9193e9d3b28ac973b27de1957273274863128833dfb99d923b might have some issues.
- Lines 10-11 import both deepspeech and stt so both would be required.
- Line 20 and 21 have a version mismatch. Line 20 should be
"model": "https://github.com/coqui-ai/STT-models/releases/download/english/coqui/v1.0.0/model.tflite",
Another little thing in the doc is that DeepSpeech can also use tflite instead of pbmm depending on how it is configured and this is how I tested DeepSpeech.
I could be wrong, but I think the imports need try
as well or it will error.
Shouldn't happen as I updated the requirements file also.
Can I close this thread?
That's true, but users would typically only need one or the other. Their requirements are likely to stray in the future as Coqui STT continues to develop.