[Bug Report]:An item with the same key has already been added. Key: xxxxxxx
ademulegun opened this issue · comments
Prerequisites
- I have searched issues to ensure it has not already been reported
Description
Whenever the consumer is started using the below code in asp.net core 7.0, the error message is displayed, " An item with the same key has already been added. Key: emailConsumer", even if you use random values. I noticed what causes the error is WithName. It throws the error when you add WithName, also when you don't add it
var provider = services.BuildServiceProvider();
var bus = provider.CreateKafkaBus();
bus.StartAsync().GetAwaiter().GetResult();
However, if you run the consumer using the below code in asp.net core 7.0, it does not throw any error, but does not consume any message published:
var kafkaBus = app.Services.CreateKafkaBus();
var lifetime = app.Services.GetRequiredService<IHostApplicationLifetime>();
lifetime.ApplicationStarted.Register(() => kafkaBus.StartAsync(lifetime.ApplicationStopped));
Steps to reproduce
Step1:
using Conflux_NotificationService.Core.Common.Options;
using Conflux_NotificationService.Core.Features.SendMail;
using KafkaFlow;
using KafkaFlow.Serializer;
using KafkaFlow.TypedHandler;
using Microsoft.Extensions.DependencyInjection;
namespace Conflux_NotificationService.Core.Extensions;
public static class KafkaBootstrap
{
private const string TopicName = "send-mail-to-tenants";
public static void BootStrapKafka(this IServiceCollection services)
{
var kafkaOptions = services.GetOptions(nameof(KafkaOptions));
services.AddKafkaFlowHostedService(
kafka => kafka
.UseConsoleLog()
.AddCluster(cluster =>
{
cluster.WithBrokers(new[] { $"{kafkaOptions.Url}:{kafkaOptions.Port}" })
.AddConsumer(consumer => consumer
.Topic(TopicName)
.WithGroupId("emailNotification")
.WithName("emailConsumer")
.WithBufferSize(100)
.WithWorkersCount(10)
.WithAutoOffsetReset(AutoOffsetReset.Earliest)
.AddMiddlewares(middleware => middleware
.AddSerializer<JsonCoreSerializer>()
.AddTypedHandlers(h=> h.AddHandler<MailRequestHandler>())));
})
);
var provider = services.BuildServiceProvider();
var bus = provider.CreateKafkaBus();
bus.StartAsync().GetAwaiter().GetResult();
}
}
Step2:
public sealed record MailRequestMessage
{
public string To { get; set; } = default!;
public string From { get; set; } = default!;
public string Subject { get; set; } = default!;
public string Body { get; set; } = default!;
}
public sealed class MailRequestHandler : IMessageHandler
{
private readonly IMediator _mediator;
private readonly ILogger _logger;
public MailRequestHandler(IMediator mediator, ILogger logger)
{
_mediator = mediator;
_logger = logger;
}
public async Task Handle(IMessageContext context, MailRequestMessage message)
{
_logger.LogInformation("Entered the MailRequestHandler");
var emailCommand = await _mediator.Send(new SendEmailCommand(message.From,
message.To,
message.Subject,
message.Body));
_logger.LogInformation("Exiting the MailRequestHandler");
await Task.CompletedTask;
}
}
Expected behavior
My expectation is when the message is published, the message handler should pick up the message
Actual behavior
System.ArgumentException: An item with the same key has already been added. Key: emailConsumer
at System.Collections.Generic.Dictionary2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary2.Add(TKey key, TValue value)
at KafkaFlow.Consumers.ConsumerAccessor.KafkaFlow.Consumers.IConsumerAccessor.Add(IMessageConsumer consumer)
at KafkaFlow.KafkaBus.StartAsync(CancellationToken stopCancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at Program.
[10:12:06 INF] API Shutting down complete
KafkaFlow version
v2.4.0
Hello @ademulegun, I've been looking into this issue, but I'm unable to reproduce it locally.
Could you show me the way you are configuring your producer? I've set it up the consumer similarly as you, and the producer like on the documentation, and it seems to function without issues.
services.AddKafka(
kafka => kafka
.UseConsoleLog()
.AddCluster(cluster =>
{
cluster.WithBrokers(new[] { $"{kafkaOptions.Url}:{kafkaOptions.Port}" })
.CreateTopicIfNotExists(EmailTopicName, 1, 1)
.AddProducer("publish-email",
producer => producer
.DefaultTopic(EmailTopicName)
.AddMiddlewares(middleware => middleware.AddSerializer()))
.AddConsumer(consumer => consumer
.Topic(ChargeBackConsumerWebTopicName)
.WithName($"{ChargeBackConsumerWebTopicName}--{Guid.NewGuid().ToString()}")
.WithGroupId("chargeBackConsumerWebMessage")
.WithBufferSize(100)
.WithWorkersCount(10)
.WithAutoOffsetReset(AutoOffsetReset.Earliest)
.AddMiddlewares(middleware => middleware.AddSerializer<JsonCoreSerializer>()
.AddTypedHandlers(h=>h.AddHandler<ChargeBackHandler>())))
.AddConsumer(consumer => consumer
.Topic(ChargeBackConsumerUssdTopicName)
.WithName($"{ChargeBackConsumerUssdTopicName}--{Guid.NewGuid().ToString()}")
.WithGroupId("chargeBackConsumerUssdMessage")
.WithBufferSize(100)
.WithWorkersCount(10)
.WithAutoOffsetReset(AutoOffsetReset.Earliest)
.AddMiddlewares(middleware => middleware.AddSerializer<JsonCoreSerializer>()
.AddTypedHandlers(h=>h.AddHandler<ChargeBackHandler>())))
.AddConsumer(consumer => consumer
.Topic(ChargeBackConsumerPosTopicName)
.WithName($"{ChargeBackConsumerPosTopicName}--{Guid.NewGuid().ToString()}")
.WithGroupId("chargeBackConsumerPosTopicName")
.WithBufferSize(100)
.WithWorkersCount(10)
.WithAutoOffsetReset(AutoOffsetReset.Earliest)
.AddMiddlewares(middleware => middleware.AddSerializer<JsonCoreSerializer>()
.AddTypedHandlers(h=>h.AddHandler<ChargeBackHandler>())))
.AddConsumer(consumer => consumer
.Topic(TransactionsConsumerWebTopicName)
.WithName($"{TransactionsConsumerWebTopicName}--{Guid.NewGuid().ToString()}")
.WithGroupId("transactionsConsumerWebTopicName")
.WithBufferSize(100)
.WithWorkersCount(10)
.WithAutoOffsetReset(AutoOffsetReset.Earliest)
.AddMiddlewares(middleware => middleware.AddSerializer<JsonCoreSerializer>()
.AddTypedHandlers(h=>h.AddHandler<TransactionMessageHandler>())))
.AddConsumer(consumer => consumer
.Topic(TransactionsConsumerPosTopicName)
.WithName($"{TransactionsConsumerPosTopicName}--{Guid.NewGuid().ToString()}")
.WithGroupId("transactionsConsumerPosTopicName")
.WithBufferSize(100)
.WithWorkersCount(10)
.WithAutoOffsetReset(AutoOffsetReset.Earliest)
.AddMiddlewares(middleware => middleware.AddSerializer<JsonCoreSerializer>()
.AddTypedHandlers(h=>h.AddHandler<TransactionMessageHandler>())))
.AddConsumer(consumer => consumer
.Topic(TransactionsConsumerUssdTopicName)
.WithName($"{TransactionsConsumerUssdTopicName}--{Guid.NewGuid().ToString()}")
.WithGroupId("transactionsConsumerUssdTopicName")
.WithBufferSize(100)
.WithWorkersCount(10)
.WithAutoOffsetReset(AutoOffsetReset.Earliest)
.AddMiddlewares(middleware => middleware.AddSerializer<JsonCoreSerializer>()
.AddTypedHandlers(h=>h.AddHandler<TransactionMessageHandler>())));
})
);
services.AddTransient(typeof(IBrokerPublisher<>), typeof(BrokerPublisher<>));
Hi @ademulegun .
When using a Hosted Service by calling AddKafkaFlowHostedService, the KafkaBus will be started by the Hosted Service. You can check the implementation of the Hosted Service here.
This means that when using AddKafkaFlowHostedService you don't need to manually start the KafkaBus so you can remove this part of your code:
var provider = services.BuildServiceProvider();
var bus = provider.CreateKafkaBus();
bus.StartAsync().GetAwaiter().GetResult();Otherwise, it will duplicate the registration of the consumers and fail when trying to register a consumer with a name that has already been registered.
Hi @ademulegun,
If you're still experiencing this issue or have any additional information to share, please feel free to let us know.
However, if we don't receive any updates or feedback from you within the next 5 business days, we may need to consider closing this issue. Please understand that this is not a final decision, and you can always reopen the issue or create a new one in the future.
We appreciate your contribution to our project and look forward to hearing from you soon. If you have any questions or need further assistance, don't hesitate to reach out.
Hi @ademulegun ,
We will be closing this issue since the question has been answered. Please feel free to respond to this message or reopen the issue if you'd like to continue the discussion or if you've encountered any new developments related to it.
Thank you for your contribution.