Upserts to mongodb some time fail

[Artem Sidorkin]

This code:

  public void addOrder(Long customerId, Long orderId, Money orderTotal) {
      BasicDBObject orderInfo = new BasicDBObject()
              .append("orderId", orderId)
              .append("orderTotal", orderTotal.getAmount());

      collection().findOneAndUpdate(new BasicDBObject("_id", customerId),
              new BasicDBObject("$set", new BasicDBObject("orders." + orderId, orderInfo)),
              upsertOptions());
  }

Can produce following error:

Caused by: com.mongodb.MongoCommandException: Command failed with error 11000 (DuplicateKey): 'E11000 duplicate key error collection: customers_orders.customers index: _id_ dup key: { _id: 1 }' on server localhost:27017. The full response is {"ok": 0.0, "errmsg": "E11000 duplicate key error collection: customers_orders.customers index: _id_ dup key: { _id: 1 }", "code": 11000, "codeName": "DuplicateKey", "keyPattern": {"_id": 1}, "keyValue": {"_id": 1}}
	at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:175)
	at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:359)
	at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:280)
	at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:100)
	at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:490)
	at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:71)
	at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:255)
	at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:202)
	at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:118)
	at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:110)
	at com.mongodb.internal.operation.CommandOperationHelper$13.call(CommandOperationHelper.java:712)
	at com.mongodb.internal.operation.OperationHelper.withReleasableConnection(OperationHelper.java:620)
	at com.mongodb.internal.operation.CommandOperationHelper.executeRetryableCommand(CommandOperationHelper.java:705)
	at com.mongodb.internal.operation.CommandOperationHelper.executeRetryableCommand(CommandOperationHelper.java:697)
	at com.mongodb.internal.operation.BaseFindAndModifyOperation.execute(BaseFindAndModifyOperation.java:69)
	at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:195)
	at com.mongodb.client.internal.MongoCollectionImpl.executeFindOneAndUpdate(MongoCollectionImpl.java:753)
	at com.mongodb.client.internal.MongoCollectionImpl.findOneAndUpdate(MongoCollectionImpl.java:733)
	at io.eventuate.examples.tram.ordersandcustomers.orderhistoryservice.domain.CustomerViewRepository.addOrder(CustomerViewRepository.java:73)
	at io.eventuate.examples.tram.ordersandcustomers.orderhistoryservice.domain.OrderViewRepositoryTest.lambda$findByIdTest$0(OrderViewRepositoryTest.java:34)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
	at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1618)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

reproduced on updated mongo (4.2.12)

https://stackoverflow.com/questions/29305405/mongodb-impossible-e11000-duplicate-key-error-dup-key-when-upserting

Looks like the only solution is to repeat insert operation.

It can be also actual for other examples.

[Chris Richardson]

5 x why.

Please provide more context:

• What test is failing?
• What is the sequence of Eventuate events (e.g. as shown by logging) that cause this scenario to occur? Presumably two event handlers are executing concurrently.

[Artem Sidorkin]

@cer

What test is failing?`

I caught it when writing queries using MongoClient.

What is the sequence of Eventuate events (e.g. as shown by logging) that cause this scenario to occur? Presumably two event handlers are executing concurrently.

In theory creating 2 orders in the same time can produce that error: https://github.com/eventuate-tram-examples/eventuate-tram-examples-micronaut-customers-and-orders/blob/master/end-to-end-tests/src/test/java/io/eventuate/examples/tram/ordersandcustomers/endtoendtests/CustomersAndOrdersE2ETest.java#L78-L82

5 x why.

There is race condition because of concurrent update of orders array of user in mongodb.
upsert does not prevent from it.

2 concurrent threads tries to add order to customer:

{
id: 1,
name: "Main Customer",
orders: []
}

{id : 1, orderTotal : 2.0}
{id : 2, orderTotal : 3.0}

upsert option prevents only from overwriting, but produces error in case of concurrent update.
It works just like JPA optimistic locking.

I reproduced it here: https://github.com/eventuate-tram-examples/eventuate-tram-examples-micronaut-customers-and-orders/pull/29/files#diff-8ff3767511223c5af33f8ff52662af599abc762440887e5955f793621ca518adR31