microsoft / kernel-memory

RAG architecture: index and query any data using LLM and natural language, track sources, show citations, asynchronous memory patterns.

Home Page:https://microsoft.github.io/kernel-memory

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

SimpleQueues: After the handle exception, it will continue to be executed repeatedly

KiisMyGun opened this issue · comments

Expectation: There is a frequency limit

public void OnDequeue(Func<string, Task<bool>> processMessageAction)
    {
        this.Received += async (sender, args) =>
        {
            try
            {
                this._log.LogInformation("Message received");

                string message = await this._fileSystem.ReadFileAsTextAsync(this._queueName, "", $"{args.MessageId}{FileExt}").ConfigureAwait(false);
                bool success = await processMessageAction.Invoke(message).ConfigureAwait(false);
                if (success)
                {
                    await this.DeleteMessageAsync(args.MessageId).ConfigureAwait(false);
                }
                else
                {
                    this._log.LogWarning("Message '{0}' processing failed, putting message back in the queue", args.MessageId);
                    this.UnlockMessage(args.MessageId);
                }
            }
#pragma warning disable CA1031 // Must catch all to handle queue properly
            catch (Exception e)
            {
                // Exceptions caught by this block:
                // - message processing failed with exception
                // - failed to delete message from disk
                this._log.LogWarning(e, "Message '{0}' processing failed with exception, putting message back in the queue", args.MessageId);
                this.UnlockMessage(args.MessageId);
            }
#pragma warning restore CA1031
        };
    } 

image

return (false, pipeline);
image

yes I've noticed that a few times with SimpleQueues, it's a known issue. While we might eventually fix it, SimpleQueues is really meant just for testing and debugging and I would not rely on it for anything more than tests.

If you need a reliable orchestration queue I strongly recommend RabbitMQ, which you can run locally too, or Azure Queues (which runs locally with the storage emulator). If you're looking for an embedded solution, I think it would be nice introducing something based on SQLite or similar.