Giters

xebia-functional / xef

Building applications with LLMs through composability, in Kotlin, Scala, ...

Home Page:https://xef.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

HuggingFace Transformers - Know How

anamariamv opened this issue · comments

commented

Hugging Face Transformers is an open-source framework for deep learning created by Hugging Face. It provides APIs and tools to download state-of-the-art pre-trained models and further tune them to maximize performance. These models support common tasks in different modalities, such as natural language processing, computer vision, audio, and multi-modal applications.

Goals:

See what options we have to integrate transformers capabilities in xef.ai. Python bindings? Maybe creating our own native JVM version?
Understand what training and fine-tuning models mean in the context of xef.ai.

commented

Good day

Are there any options available in xef to use HuggingFace Transformers instead of the OpenAI API? In the context of the conversation() block.

As far as I can tell, it's internally configured to use OpenAI. Is it possible to use Hugging Face models instead? Is that work in progress?

We're eager to use xef in production, but the high latency with the OpenAI requests is quite an inconvenience ... that's why we're curious about alternative options that xef has to offer

commented

Hi @ff137!

We're working on adding more flexibility around the OpenAI API backend, allowing other servers with a compatible API to be used for particular requests. All this effort is initially planned for the Xef server, and its functionality can be shown in the following diagram:

sequenceDiagram
    conversation()->>+OpenAIClient: SDK Call
    OpenAIClient->>+XefServer: (OpenAI host = Xef Server)
    XefServer->>OpenAIRest: Request
    OpenAIRest-->>XefServer: Response
    XefServer-->>-OpenAIClient: 
    OpenAIClient-->>-conversation():

This approach will also allow Xef users to implement custom mappings for servers with an API different from OpenAI.

This functionality is underway, and one of the features was added in this PR #513. Example 1 shows how to use huggingface/text-embeddings-inference for embeddings given a specific model.

commented

Can we mark as completed, @anamariamv ?