Client + Model incompatibility disagnostics

[Josh Meyer]

For support and discussions, please use our Discourse forums.

If you've found a bug, or have a feature request, then please create an issue with the following information:

• Have I written custom code (as opposed to running examples on an unmodified clone of the repository):
  No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Manjaro Linux 20.2.1
• TensorFlow installed from (our builds, or upstream TensorFlow):
  pip install deepspeech-tflite in python3.9 venv
• TensorFlow version (use command below):
  v2.3.0
• Python version:
  3.9.1
• Bazel version (if compiling from source):
  NA
• GCC/Compiler version (if compiling from source):
  NA
• CUDA/cuDNN version:
  NA
• GPU model and memory:
  NA
• Exact command to reproduce:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -U pip
$ pip install deepspeech-tflite
$ wget https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/deepspeech-0.9.3-models.pbmm
$ wget https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/audio-0.9.3.tar.gz
$ tar -xvzf audio-0.9.3.tar.gz
$ deepspeech --model deepspeech-0.9.3-models.pbmm --audio audio/2830-3980-0043.wav

Please describe the problem clearly. Be sure to convey here why it's a bug or a feature request.

I know this incompatibility is documented on readthedocs here [1], but I still think my suggestions are valid.

The incompatibility between client and model WRT TensorFlow Lite doesn't produce a clear error message, and It's impossible AFAIK to know whether you installed deepspeech-tflite or deepspeech or deepspeech-gpu after the fact.

I would suggest (1) that deepspeech --version produces information on which kind of client was installed, and also (2) a clear error message if you try to mix incompatible models and clients

Include any logs or source code that would be helpful to diagnose the problem. For larger logs, link to a Gist, not a screenshot. If including tracebacks, please include the full traceback. Try to provide a reproducible test case.

(venv) [josh@macbook ~]$ deepspeech --model deepspeech-0.9.3-models.pbmm --audio audio/2830-3980-0043.wav 
Loading model from file deepspeech-0.9.3-models.pbmm
TensorFlow: v2.3.0-6-g23ad988
DeepSpeech: v0.9.3-0-gf2e9c85
ERROR: Model provided has model identifier 'u/3�', should be 'TFL3'

Error at reading model file deepspeech-0.9.3-models.pbmm
Traceback (most recent call last):
  File "/home/josh/venv/bin/deepspeech", line 8, in <module>
    sys.exit(main())
  File "/home/josh/venv/lib/python3.9/site-packages/deepspeech/client.py", line 119, in main
    ds = Model(args.model)
  File "/home/josh/venv/lib/python3.9/site-packages/deepspeech/__init__.py", line 38, in __init__
    raise RuntimeError("CreateModel failed with '{}' (0x{:X})".format(deepspeech.impl.ErrorCodeToErrorMessage(status),status))
RuntimeError: CreateModel failed with 'Failed to initialize memory mapped model.' (0x3000)

[lissyx]

It's impossible AFAIK to know whether you installed deepspeech-tflite or deepspeech or deepspeech-gpu after the fact.

pip list show shows deepspeech, deepspeech-gpu or deepspeech-tflite

(1) that deepspeech --version produces information on which kind of client was installe

Previously, TFLite runtime would advise itself on stdout or stderr, it looks like it is not the case anymore.
Technically, exposing into --version is doable, but my fear is that people will start to want to have that and others informations exposed, while we want to keep the lib as agnostic as possible.

We already had people trying to do nasty things on CUDA and asking for us to expose CUDA infos ...

(2) a clear error message if you try to mix incompatible models and clients

I'm not sure what room we have here for improvement without doing hacky things for a low outcome: we don't open the file ourselves, so when it fails to open it's already to late.

Here

DeepSpeech/native_client/tflitemodelstate.cc

Lines 166 to 170 in 7b2eeb6

 fbmodel_ = tflite::FlatBufferModel::BuildFromFile(model_path); 
 if (!fbmodel_) { 
   std::cerr << "Error at reading model file " << model_path << std::endl; 
   return DS_ERR_FAIL_INIT_MMAP; 
 } 

we detect the error from TFLite runtime, but if we want to provide more meaningful error, we would have to:

• open ourselves the file
• start some parsing (painful to maintain potentially)
• introduce a new error code (easy)

And from there, what better error message would you build?

I fail to get the usecase to address here, people are supposed to know what they have installed, we have different package names for the different runtimes so a package manager list operation is enough to know what runtime runs, and the error message indicates it's not the correct file. Yes, maybe it could be nicer to have a better error message, but I'm not really sure it's super critical as of now, given the downsides it pulls ...

[lissyx]

Maybe leverage ErrorReporter ? https://github.com/tensorflow/tensorflow/blob/r2.3/tensorflow/lite/model_builder.cc#L63-L73