Unexpected Behavior in NATS DeliverGroup: Load Balancing Feature Not Working as Expected with Same Queue Group

[Savorboard]

Defect

The queue group name which, if specified, is then used to distribute the messages between the subscribers to the consumer. This is analogous to a queue group in core NATS.

If more subscribers are added to the same queue name, they become a queue group, and only one randomly chosen subscriber of the queue group will consume a message each time a message is received by the queue group. Such distributed queues are a built-in load balancing feature that NATS provides.

Based on the NATS documentation about DeliverGroup, the Delivery Group is supposed to have a load balancing feature. After testing, I found that the queue group parameter did not work as expected.

The following example code is used to reproduce the process: creating three consumers within a single Stream, all sharing a queue group. The message should be received by one of the three consumers, but in reality, all three receive it.

Versions of NATS.Client and nats-server:

NATS-Server: V2.6.5
NATS-Client: V1.0.3

Steps or code to reproduce the issue:

public IJetStream _js;
public string _subject = "test.subject";
public string _stream = "test";

void Main()
{
	var _connectionFactory = new ConnectionFactory();
	var connection = _connectionFactory.CreateConnection("nats://localhost:4222");

	_js = connection.CreateJetStreamContext();


	//add stream
	var builder = StreamConfiguration.Builder()
					  .WithName(_stream)
					  .WithNoAck(false)
					  .WithRetentionPolicy(RetentionPolicy.Interest)
					  .WithStorageType(StorageType.Memory)
					  .WithSubjects(_subject);
	var jsm = connection.CreateJetStreamManagementContext();
	jsm.AddStream(builder.Build());

	//consumer
	var pso = PushSubscribeOptions.Builder()
					.WithStream(_stream)
					.WithConfiguration(ConsumerConfiguration.Builder()
						.WithDeliverPolicy(DeliverPolicy.All)
						.WithAckPolicy(AckPolicy.Explicit)
						.Build())
					.Build();
	for (int i = 0; i < 3; i++)
	{
		var group = "samegroup";
		Task.Run(() =>
		{
                        // `queue: group` is the queue group
			_js.PushSubscribeAsync(subject: _subject, queue: group, SubscriptionMessageHandler, false, pso);
		});
	}


	//producer
	Task.Run(() =>
	{
		while (true)
		{
			var timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
			Thread.Sleep(2000);
			Publish(timestamp.ToString());
		}
	});

	Console.ReadLine();
}

void SubscriptionMessageHandler(object sender, MsgHandlerEventArgs e)
{
	Console.WriteLine(Encoding.UTF8.GetString(e.Message.Data));
	e.Message.Ack();
}

void Publish(string body)
{
	var msg = new Msg(_subject, Encoding.UTF8.GetBytes(body));
	_js.Publish(msg);
}

Expected result:

1681534945
1681534947
1681534949

Actual result:

1681534945
1681534945
1681534945
1681534947
1681534947
1681534947
1681534949
1681534949
1681534949

[Scott Fauerbach]

@yang-xiaodong Your subscriptions are ephemeral which means they will be different consumers. Since this is JetStream, the consumer must be durable. I probably should add exception the subscription if you try to make a queue'd one without a durable.

[Scott Fauerbach]

Also, you are on a pretty old server, I would move to 2.9.15 then 2.9.16 when it becomes available.