Mongock migration for MongoDB on CosmosDB fails

[hans009]

Description

Spring Boot Application fails to start when running change unit within a transaction

PRIORITY

NORMAL

Version and environment

Mongock

• 5.1.5
• Spring Boot, MongoDB (on CosmosDB with 4.2 API), JPA 2.2.3
• Annotation approach is used

Environment

• Spring Boot 2.7.11
• Spring Data MongoDB 3.4.11
• JPA 2.2.3
• Hibernate 5.6.15.Final
• MongoDB driver 4.6.1 (org.mongodb.*)
• Infrastructure: AKS

Steps to Reproduce

1. Create migration class and add a changeunit which adds a document to a collection (can be empty)
2. Make sure a MongoTransactionManager is used
3. Deploy to Azure AKS and wait till the execution runs

Behaviour

Expected behavior: The new document is added to the collection on application start

Actual behavior: The whole thing is running in a spring boot application which primarily uses JPA for persisting data. MongoDB was added recently on top. Unfortunately when deployed to Azure the application doesn't start due to the error attached below.

How often the bug happens: every time, 100%

Link to repository using Mongock

n/a

Additional context

Following error occurs:

Caused by: com.mongodb.MongoWriteException: Write operation error on server theserver.mongo.cosmos.azure.com:10255. Write error: WriteError{code=2, message='Error=2, Details='Response status code does not indicate success: BadRequest (400); Substatus: 1101; ActivityId: ca08a5ad-e6e7-412e-81c1-fb39da2d4aeb; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 1101; ActivityId: ca08a5ad-e6e7-412e-81c1-fb39da2d4aeb; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 1101; ActivityId: ca08a5ad-e6e7-412e-81c1-fb39da2d4aeb; Reason: (Message: {"Errors":["Transaction is not active"]}
ActivityId: ca08a5ad-e6e7-412e-81c1-fb39da2d4aeb, Request URI: /apps/7dd4ccb4-27e8-47e0-9b06-f2e87b81bab2/services/e77183bd-436f-46c6-86b2-c5b0245e4334/partitions/0b624621-b727-49c5-9ca0-31715b4f71ab/replicas/133401303155692792p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.17763 cosmos-netstandard-sdk/3.18.0);););', details={}}.
        at com.mongodb.client.internal.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:1018)
        at com.mongodb.client.internal.MongoCollectionImpl.executeUpdate(MongoCollectionImpl.java:994)
        at com.mongodb.client.internal.MongoCollectionImpl.updateOne(MongoCollectionImpl.java:591)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

The change unit is like this:

@ChangeUnit(id = "users-initialization", order = "001")
@RequiredArgsConstructor
@Slf4j
public class InitialSetupMigration {

    private final MongoTemplate template;

    @Execution
    public void changeSet() {
        // @Execution is run within transaction
        Authority userAuthority = createUserAuthority();
        userAuthority = template.insert(userAuthority);
    }
...

Writing documents generally is working, so the connections string is correct. Running a MongoDB locally in Docker or with testcontainers is working. Mongock is using the MongoTransactionManager when available and always tries to run a changeunit within an own transaction. If I add transactional = false to @ChangeUnit annotation it works on Azure as well. I've read that CosmosDB is not supporting multidocument transactions on sharded collections, so I created the collection as unsharded collection with no success either. When I disable the Mongock migration the application runs (on Azure) and I can create documents in mongodb. It looks like there is not transaction available on Azure, how is this possible?
Any help is very much appreciated

[hans009]

Hello again, I found out that it's related to MongoDB on CosmosDB (non-vCore) inability to span transactions accross multiple collections.
The issue is that after executing a change unit (e.g. adding something to collection_one) the mongockChangeLog collection should get updated within the same transaction and thus fails due to that limitation.
So maybe you can add to your CosmosDB documentation that in the non-vCore mode you cannot use native transactions in that scenario but instead have to implement the rollback logic in @RollbackExecution method yourself

[Oliver Santana]

Hello @hans009 , thanks for raising the issue and for your workaround. We'll take it into account to update Mongock's documentation.