vlnk/whisper.el

whisper.el

Speech-to-Text interface for Emacs using OpenAI’s whisper speech recognition model. For the inference engine it uses the awesome C/C++ port whisper.cpp that can run on consumer grade CPU (without requiring a high end GPU).

You can capture audio with your local input device (microphone) or choose a media file on disk in your local language, and have the transcribed text pasted to your Emacs buffer (optionally after translating to English). This runs offline without having to use non-free cloud service for decent result (though result quality of whisper varies widely depending on language, see below).

Install and Usage

Aside from a C++ compiler (to compile whisper.cpp), the system needs to have FFmpeg for recording audio.

You can install whisper.el by cloning this repo somewhere, and then use it like:

(use-package whisper
  :load-path "path/to/whisper.el"
  :bind ("C-H-r" . whisper-run)
  :config
  (setq whisper-install-directory "/tmp/"
        whisper-model "base"
        whisper-language "en"
        whisper-translate nil))

You will use these functions:

whisper-run: Toggle between recording from your microphone and transcribing
whisper-file: Same as before but transcribes a local file on disk

Invoking whisper-run with a prefix argument (C-u) has the same effect as whisper-file.

Both of these functions will automatically compile whisper.cpp dependency and download language model the first time they are run. When recording is in progress, invoking them stops it and starts transcribing. Otherwise if compilation, download (of model file) or transcription job is in progress, calling them again cancels that.

Note for MacOS users: If whisper.el is failing silently, it might be because Emacs doesn’t have the permission to use the Mic. Follow the recipe in wiki to grant it explicitly.

Variables

whisper-install-directory: Location where whisper.cpp will be installed. Default is ~/.emacs.d/.cache/.
whisper-language: Specify your spoken language. Default is en. For all possible short-codes: see here. You can also set it to auto to allow whisper.cpp to infer the language from first 30 seconds of audio. How well whisper works will vary depending on the language. Some scores could be found in the original paper, or here.
whisper-model: Which language model to use. Default is base. Values are: tiny, base, small, medium, large-v1, large. Bigger models are more accurate, but takes more time and more RAM to run (aside from more disk space and download size), see: resource requirements. Note that these come with .en variants that might be faster, but are for English only.
whisper-translate: Default nil means transcription output language is same as spoken language. Setting it to t translates it to English first.
whisper-insert-text-at-point: By default whisper.el inserts the transcribed text at the point where whisper-run or whisper-file was invoked. But if you set this to nil, the text will be displayed in a separate buffer instead.
whisper-use-threads: Default nil means let whisper.cpp choose appropriate value (which it sets with formula min(4, num_of_cores)). If you want to use more than 4 threads (as you have more than 4 cpu cores), set this number manually.
whisper-recording-timeout: Default is 300 seconds. We do not want to start recording and then forget. The intermediate temporary file is stored in uncompressed wav format (roughly 4.5mb per minute but can vary), they can grow and fill disk even if /tmp/ is used for it by default.
whisper-show-progress-in-mode-line: By default, progress level of running job in whisper.cpp is shown in the mode line.
whisper-enable-speed-up: Default is nil. This can supposedly speed up transcribing up to 2x, at the expense of some accuracy loss. You should experiment if it works for you, specially when using larger models. Note: As of Nov 2023, setting this option does nothing. It was recently disabled by upstream “temporarily” due to bugs, we will see if it comes back.
whisper-quantize: Whether to quantize the model (default nil). Non-nil valid values are: q4_0, q4_1, q5_0, q5_1, q8_0. For an explanation of what quantization means, see below. If it’s defined, whisper.el will automatically quantize the model before using that.
whisper-install-whispercpp: By default the installation of whisper.cpp is done automatically. If you are on a platform where our automatic install fails, but you are able to do so manually at whisper-install-directory, you can set this to =’manual= to ensure we don’t try and fail to install it automatically. Also if you are planning to not use whisper.cpp at all by overriding whisper-command (see below), you can just set this to nil to ensure no whisper.cpp related runtime checks and downloads will be performed.

Additionally, depending on your input device and system you will need to modify these variables to get recording to work:

whisper--ffmpeg-input-format: This is what you would pass to the -f flag of FFmpeg to input, to record audio. Default is pulse on Linux, avfoundation on OSX and dshow on Windows.
whisper--ffmpeg-input-device: This is what you would pass to the -i flag of FFmpeg to record audio, like hw:0,2 or something. There is no default (unless you are using pulseaudio or pipewire-pulse in that case it’s default source) so this will likely need to be set. For MacOS users, the wiki contains a recipe that lets you set this interactively: see here.

Pulseaudio and PipeWire users who haven’t further configured their default source may find that results are better when at least echo cancel filter is enabled (by loading relevant module). Then you could either set that as the default source (using e.g. pactl) or just use that source’s name in whisper--ffmpeg-input-device. Furthermore, the following programs could be used to improve quality of audio recording in general:

Easy Effects
NoiseTorch
DeepFilterNet (see also PipeWire integration)

Hooks

There are couple of hooks provided for user defined actions:

whisper-pre-process-hook: Functions here are run before anything else. Helpful to make ensure suitable condition to run whisper (e.g. check if buffer is read-only).

whisper-post-process-hook: If you want to do some text formatting or transformation on the whisper output, add a function here. For example, the default output is one big line of text. If you want to do something like adding a paragraph break every N sentences, you could do something like:

(defun whisper--break-sentences (n)
  "Put a paragraph break every N sentences."
  (save-excursion
    (goto-char (point-min))
    (catch 'return
      (while t
        (dotimes (_ n)
          (forward-sentence 1)
          (when (eobp) (throw 'return nil)))
        (insert "\n\n")
        (when (= (char-after) ?\ )
          (delete-horizontal-space))))))

(add-hook 'whisper-post-process-hook
          (lambda ()
            (whisper--break-sentences 5))) ;; add a paragraph break every 5 sentences

Performance Guide for Advanced Users

By default, whisper.cpp performance on CPU is likely good enough for most people and most use cases. However if it’s not good enough for you, here are some things you could do:

Update whisper.cpp

The upstream whisper.cpp is continuously improving. If you are using an old version, updating whisper.cpp is the first thing you could try. Simplest way to do that is to delete your the whisper.cpp installation folder and re-run the command, which will reinstall from latest commit.

Quantize the model

Quantization is a technique to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types. This sacrifices precision for resource efficiency. The idea is that quantized version of a bigger model may afford you to use it (if you are RAM constrained e.g.) with some penalty or accuracy, while still being more accurate hopefully than the smaller model you would be using otherwise.

Re-compile whisper.cpp for hardware acceleration

Offloading the encoder inference to hardware or optimised external libraries may result in speed-up. There are options to use: Core ML (for Apple hardware), cuBLAS (for NVIDIA GPU), OpenVINO (Intel CPU/GPU), CLBlast (for GPUs that support OpenCL), OpenBLAS (an optimised matrix processing library for CPU). Consult whisper.cpp README for how to re-compile whisper.cpp to enable those.

Use something other than whisper.cpp

If you think there is something else you want to use, you have the option to override the whisper-command function definition (or define an overriding advice). This function takes a path to input audio file as argument, and returns a list denoting the command to be run instead of whisper.cpp. You can use the variables described above to devise the command. The wiki contains a recipe that shows how to use whisper-ctranslate2 with whisper.el. This client is compatible to OpenAI’s original one, so porting the recipe to use the original client should be possible.

Note that when you are using something other than whisper.cpp, the onus is on you to make sure the target program is properly installed and relevant model files for it are downloaded beforehand. We don’t support anything other than whisper.cpp so any problems integrating them with whisper.el that’s specific to those software may strain our ability to address.

Caveats

Whisper is open-source in the sense that weights and the engine source is available. But training data or methodology is not.
Real time transcribing is probably not feasible with it yet. The accuracy is better when it has a bigger window of surrounding context. Plus it would need beefy hardware to keep up, possibly using a smaller model. There is some interesting activity going on at whisper.cpp upstream, but in the end I don’t see the appeal of that in my workflow (yet).

vlnk / whisper.el