transformerlab / transformerlab-app

Open Source Application for Advanced LLM Engineering: interact, train, fine-tune, and evaluate large language models on your own computer.

Home Page:https://transformerlab.ai/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Check for supported file formats when displaying available inference engines

dadmobile opened this issue · comments

Specifically MLX only supports some weight file formats (safetensors and nfz I think?). We currently only check architecture which means you sometimes get a "No safetensors for..." error when trying to run a model with MLX.

There are several possible ways to address:

  • check weight file formats (currently there is a formats array in model gallery)
  • use allow_patterns to see if there's a supported file type (lots of reasons this might not work)
  • take advantage if we add MLX field to models that says minimum version required to support like transformers (separate issue)
  • create some other way for a plugin to take a model and return if it's supported (kind of ugly but more flexible)