Can tgi surpport non tokenizer model？

[Strive-for-excellence]

Feature [request]

Recently, multimodal large models based on the Transformer architecture have emerged one after another. Can text-generate-inference provide some support? For example, a feasible solution is for text-generate-inference to only support the inference part, while encoding and decoding are handled by the user. When making a request, input token IDs and return token IDs, rather than text. Examples of such projects include multimodal models like (https://github.com/2noise/ChatTTS) and (https://github.com/RVC-Boss/GPT-SoVITS).

Motivation

none

Your contribution

none