xebia-functional / xef

Building applications with LLMs through composability, in Kotlin, Scala, ...

Home Page:https://xef.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Update Memory storing

Montagon opened this issue · comments

This issue tries to address two things in memory:

  1. Memory class is managing now only 1 message but this is a problem in our implementation. Chat completion is receiving a list of messages and when storing the messages in the memory we need to add the timestamp to every message. That is not correct because getting the message from the conversationId we are sorting by timestamp, and several messages can be the same timestamp
  2. The memory could store the first simple implementation of tracing. We can add information about the time it takes for the IA to respond and the number of tokens of the call

Memory could be something like this:

data class Memory(
  val conversationId: ConversationId,
  val timestamp: Long,
  val request: List<Message>,
  val aiResponse: List<Message>,
  val responseTimeInMillis: Long,
  val tokens: Int
)

With this approach, we have:

  1. Exactly the tokens in the call, we don't need to store the approximated tokens for every message
  2. The request and the response in every call as the user did
  3. We can calculate the money spent on each call using the real tokens
  4. We can have an endpoint in the server for getting all the information based on the conversationId

For this approach, we have to change some parts of the Xef implementation:

  1. The way to store the memory in the VectorStore (local, Postgres and Lucene)
  2. The way to get the memory from the VectorStore
  3. Changes in the PromptCalculator

It looks good, thank you!

Thanks @Montagon and @javipacheco I like this approach!

Finally, we are changing this approach. After some conversations this week with @raulraja we are going to split the 2 responsibilities in this approach for the memory

The first ticket is #466, this is only for the memory. Next week I'll create a couple issue more about the new strategy for tracing.

This new approach that we are working help us in future implementations like ReActAgent