[BUG] Summarizer should skip summarization if new message is empty

[Daniel Chalef]

Describe the bug
hey @daniel Chalef ! We're currently using metadata to store the S3 url for images, and we've noticed that when the memory chat history is sent via Langchain, the url seems to be replaced with a newline. Then the summary starts to ingest this as "The New Lines do not provide meaningful content.".

Is it possible to have the summary ignore new lines somehow, or at least not comment on it as part of the summary?

To Reproduce
For example, this is what the prompt via Langsmith will look like:

System: The Current Content contains a lighthearted exchange about financial stress. The New Lines do not provide meaningful content.
Human: chat
AI: image

Human: chat
AI: image

Human: chat

Expected behavior
Is it possible to have the summary ignore new lines somehow, or at least not comment on it as part of the summary?

Logs
Please provide your Zep server and applications logs.

Environment (please complete the following information):

• Zep version: [e.g. vX.X.X]
• Zep SDK and version: [e.g. zep-js or zep-python and vX.X.X]
• Deployment [e.g. using docker compose, to a hosted environment such as Render]

Note: The Zep server version is available in the Zep server logs at startup:
Starting zep server version 0.11.0-cbf4fe4 (2023-08-30T12:49:03+0000)

Additional context
Add any other context about the problem here.

[ksrath]

Thanks @danielchalef . What I've also noticed is that new chat messages don't seem to get summarised into the summary after 'The New Lines do not provide meaningful content.' So essentially the memory gets stuck in this state.

[Daniel Chalef]

Fixed in #287

Will be available in the next release of Zep. Likely today.

[ksrath]

hi @danielchalef - Thanks for this. I did some preliminary testing and was pretty happy with it as it seemed to have been resolved. However I have noticed that as the user continues to chat and multiple summaries are created, I see the bold text below. Just wondering if this is expected behaviour? I did do some testing on new lines (which occur as we use the metadata to store S3 links to images and video, etc.). Is this still relating to new lines? As I hadn't seen this up until we started using the metadata.

System: The current content is:
<summary here>
...Lucy signs off and the AI bids her farewell.

  **The New Lines are meaningless and do not add any relevant information to the conversation. Therefore, the Current Content remains the same.**
  AI: You're welcome, Lucy

[Daniel Chalef]

@ksrath Which LLM API provider and model are you using? Hoping to replicate the issue here.

[ksrath]

Hi @danielchalef we're using OpenAI gpt-3.5-turbo. After looking, I'm not necessarily sure it has to do with metadata usage anymore. It seems like after ~ the 4th/5th summary, there is a message along the lines of:
(No new summary provided, as the new lines do not add meaning to the existing content.)

Which I think is incorrect - the 5 (or so) chats that fell out of the chat history window have relevant content to be summarised. Let me know if you need more info.

Here is our code for saving our S3 url against our chat metadata
```

save image to metadata

        metadata = {"url": url, "type": "image"}
        get_zep_memory(chat_id).chat_memory.add_ai_message(
            message=prompt_text,
            metadata=metadata
        )

[ksrath]

Hi @danielchalef - just wondering if there's any update on this, as we're continuously seeing it as users chat more and multiple Zep summaries are being created, see below:

# Conversation history:
System: The current content does not need to be updated as the new lines of conversation are not meaningful.

Is it possible for me to query old summaries, perhaps that will help with replicating it for you?

[Daniel Chalef]

Hey @ksrath , sorry I missed your message. You can search over historical summaries for a session: https://docs.getzep.com/sdk/chat_history/search/

I'm unsure why the LLM is responding as such. Are you able to share sanitized examples of dialogs where this occurs?

[ksrath]

Hi @danielchalef , no worries. Here are some details.

model: "gpt-3.5-turbo"
message window size: 16

I was able to replicate and here are some examples of the convo history where it occurred.
See below the conversation history prior to the first summary being created:

Human: heya
AI: Hey there! How's your day going? Anything interesting happening?
AI: Hey Sez! It's Jaimee. Merry Christmas! 🎄 Hope you're having a wonderful day!
AI: Hey Sez! Just wanted to check in and see how you're doing, miss you! 😊
AI: Hey Sez! It's Jaimee. How's everything going? Miss you! 😘
AI: Hey Sez! It's Jaimee. How's everything going? Miss you! 😘
AI: Hey Sez! It's Jaimee. Just wanted to say hi and see how you're doing, miss you! 😊
Human: heya
AI: Hey there! How's your day going? Anything interesting happening?
Human: can you searhch for the closest cafe to me?
AI: I found some great options for cafes near you. Let me know if you want more details!
Human: what were they?
AI: I found a few options for cafes near you. How about we grab a cup of coffee and catch up? ☕️

Once I've replied "can you searhch for the closest cafe to me?" the first summary took in the first 6 messages, and it then looked like this:

The provided conversation does not contain meaningful information for creating a summary. Therefore, the current content remains.
Human: heya
AI: Hey there! How's your day going? Anything interesting happening?
Human: can you searhch for the closest cafe to me?
AI: I found some great options for cafes near you. Let me know if you want more details!
Human: what were they?
AI: I found a few options for cafes near you. How about we grab a cup of coffee and catch up? ☕️

I think it could occur when there's consecutive 'AI' messages without a Human message.