Improve Query Abstraction (issue/improvement)

Question

Improve Query Abstraction (issue/improvement)

amoscatelli opened this issue 3 years ago · comments

amoscatelli commented 3 years ago

Hi there,
I am trying to integrate JNoSql Document abstraction but I have some problem with the spec itself I believe.

I want to do a simple query : A and B or C and D
Where A, B, C and D are simple field predicates (es. field = value).
I am using the MongoDB driver and I noticed that if I write
where(fieldA).eq(valueA).and(fieldB).eq(valueB).or(fieldC).eq(valueC).and(fieldD).eq(valueD)
this is actually translated into : ((A and B) or C) and D
I confirmed this by printing MongoDB queries into console.
I believe the left associative strategy it's not the best to represent every possibile query scenario.
Is this intended ?
How can I do a query like (A and B) or (C and D) ?
Why don't we create a proper Predicate abstraction similar to the one from JPA specification ?
I expected a more hierarchical approach, to be able to do something like where(predicates) and predicates being a Collection of Predicate where a Predicate can be a Conjunctive (AND) or a Disjunctive (OR) or etc etc.
Also we are completely missing the possibily to UPDATE by query/condition.
I mean updating every document setting a field to a certain value where a condition is specified.
Is this intended too ?
All of this is completed supported by MongoDB.
Are there other Document databases who don't support these two features ?

I can/want to contribute on both of these requirement if we do agree on them.

I tried to subscribe to the dev mailing list but I was not accepted so far.

What do you thing about this ?

@otaviojava

Otávio Santana · Answer 1 · Thu Apr 07 2022 21:44:39 GMT+0800 (China Standard Time)

Hey, @amoscatelli it sounds like a huge challenge.
Sure, please, go ahead.
Please, let me know if you need any help.

amoscatelli · Answer 2 · Thu Apr 07 2022 22:09:37 GMT+0800 (China Standard Time)

Hi @otaviojava . Apart from the required changes I am willing to contribute with, do you already know any reason for this to be incompatible with other Document databases (for example OrangoDB?). I never worked with other document oriented databases apart from MongoDB

Otávio Santana · Answer 3 · Thu Apr 07 2022 22:34:10 GMT+0800 (China Standard Time)

In documents databases, it should be fine.
We have tests on our drivers, so it will be ok for us to test.

amoscatelli · Answer 4 · Mon Apr 11 2022 21:36:58 GMT+0800 (China Standard Time)

I started working on the spec/abstraction

https://github.com/amoscatelli/nosql

Most of the work will be in the jakarta.nosql.column.criteria package, at first

amoscatelli · Answer 5 · Tue Apr 12 2022 01:39:23 GMT+0800 (China Standard Time)

@otaviojava ok, I believe I finished the abstraction for the CriteriaQuery API.
It is largely inspired by JPA, but it is indeed only a subset of those functionalities.

Tomorrow I'll finish adding comments to every interface, then I'll make a pull request and I'll start working on the eclipse implementation.
At last, I will work on the MongoDB driver.

Please have a look and tell me if you like the direction I took.
Also notice I will integrate a sort of metamodel generator in the eclipse implementation, scanning for @entity and creating the metamodel on its own.

This is only the first part of what I am thinking of.
CriteriaUpdate and CriteriaDelete will follow.

amoscatelli · Answer 6 · Tue Apr 12 2022 01:40:31 GMT+0800 (China Standard Time)

I want to stress out tha fact that the Metamodel is a requirement to use the api.

Otávio Santana · Answer 7 · Tue Apr 12 2022 17:26:46 GMT+0800 (China Standard Time)

@amoscatelli you did not send the PR to review.

amoscatelli · Answer 8 · Tue Apr 12 2022 18:36:02 GMT+0800 (China Standard Time)

@otaviojava here you are, I was finishing the comments before making a pull request

Otávio Santana · Answer 9 · Tue Apr 12 2022 23:02:52 GMT+0800 (China Standard Time)

@amoscatelli
That is great progress.
Are you going to create the implementation in the JNoSQL?

amoscatelli · Answer 10 · Tue Apr 12 2022 23:05:26 GMT+0800 (China Standard Time)

Yes, I am going to. I wanted to know if you agreed before moving further.
What I would like NOT to do, is to update every single document driver.
I am going to take care of the MongoDB one, I hope the community will take care of the others (if any change is required indeed, I am not so sure atm).

Otávio Santana · Answer 11 · Tue Apr 12 2022 23:18:34 GMT+0800 (China Standard Time)

@amoscatelli if it will implement MongoDB
I believe that it would be better to start as an extension instead of the main API:

https://github.com/eclipse/jnosql-mapping-extension

Otávio Santana · Answer 12 · Tue Apr 12 2022 23:18:45 GMT+0800 (China Standard Time)

Please, let me know what do you think.

amoscatelli · Answer 13 · Tue Apr 12 2022 23:23:38 GMT+0800 (China Standard Time)

I am not sure I understand what you are saying.
Didn't we agree that this abstraction is compatible with all Document dbs ?
I added a method on the DocumentTemplate interface, is that wrong ?
Also, I believe MongoDB drivers will need a change since it uses left associative strategy by default.

Anyway, if you want me to start as an extension is ok for me, please guide me on how to proceed and how to test the extension as I develop it.

What do I need to do next ?

Otávio Santana · Answer 14 · Wed Apr 13 2022 00:28:49 GMT+0800 (China Standard Time)

@amoscatelli
I'm thinking 🤔
In fork; https://github.com/eclipse/jnosql-mapping-extension

create a module call: criteria
move the whole API code there
Create a MongoDB module
Implement there

What do you think?

amoscatelli · Answer 15 · Wed Apr 13 2022 18:01:57 GMT+0800 (China Standard Time)

@otaviojava
Sorry, I still don't get it.
Do you want me to fork https://github.com/eclipse/jnosql-mapping-extension ?
What do you mean by module call ?
Can you show me an example of what I am supposed to do ?

amoscatelli · Answer 16 · Wed Apr 13 2022 18:08:20 GMT+0800 (China Standard Time)

in https://github.com/eclipse/jnosql-mapping-extension I can see only db specific packages.
Where should I move the jakarta.nosql.criteria package ?

Otávio Santana · Answer 17 · Wed Apr 13 2022 18:50:05 GMT+0800 (China Standard Time)

As the first step, yes.
we can use it there to get mature and explore, even, to receive more feedback from the community, then, move it to the core API.

Otávio Santana · Answer 18 · Thu Apr 14 2022 16:52:46 GMT+0800 (China Standard Time)

@amoscatelli sorry, I did check the code again.
It makes sense to keep on the core as you did.
As soon, you create it to MongoDB we can merge both.
Please, let me know what you think.
Again, sorry for that.

amoscatelli · Answer 19 · Thu Apr 14 2022 16:59:26 GMT+0800 (China Standard Time)

Today I'll start working on it

Otávio Santana · Answer 20 · Thu Apr 28 2022 00:54:44 GMT+0800 (China Standard Time)

@amoscatelli congrats on the progress, but in the end, it should return a DocumentQuery

Otherwise, it won't be compatible with what we have.
You can use the DocumentCondition

E.g.: the (A and B) or (C and D)

DocumentCondition a = DocumentCondition.eq(Document.of("name", "Poliana"));
DocumentCondition b = DocumentCondition.eq(Document.of("name", "Otavio"));
DocumentCondition c = DocumentCondition.eq(Document.of("city", "São Paulo"));
DocumentCondition d = DocumentCondition.eq(Document.of("city", "Salvador"));
DocumentCondition andAB = DocumentCondition.and(a, b);
DocumentCondition andDC = DocumentCondition.and(c, d);
DocumentCondition finalCondition = DocumentCondition.or(andAB, andDC);

So, in this criteria API you can create this implementation:

 DocumentQuery query = new DocumentQuery() {
            @Override
            public long getLimit() {
                return 0;
            }

            @Override
            public long getSkip() {
                return 0;
            }

            @Override
            public String getDocumentCollection() {
                return null;
            }

            @Override
            public Optional<DocumentCondition> getCondition() {
                return Optional.empty();
            }

            @Override
            public List<Sort> getSorts() {
                return null;
            }

            @Override
            public List<String> getDocuments() {
                return null;
            }
        }

amoscatelli · Answer 21 · Thu Apr 28 2022 07:47:20 GMT+0800 (China Standard Time)

Are you sure ?
The new API will enable us to do things like this (I am finishing implementing this in the MongoDB driver) :

        CriteriaQuery<Person> criteriaQuery = new DefaultCriteriaQuery(Person.class);
        StringExpression<Person, Person> get = criteriaQuery.from().get(Person.NAME);
        CriteriaFunction<Person, Person, Person, Number> count = criteriaQuery.from().count();
        entityManager.executeQuery(
                criteriaQuery.select(
                        count
                ).where(
                        get.equal("Poliana")
                )
        );

Or for example I can even have a SUM operator

        CriteriaQuery<Person> criteriaQuery = new DefaultCriteriaQuery(Person.class);
        NumberExpression<Person, Person, Integer> get = criteriaQuery.from().get(Person.AGE);
        entityManager.executeQuery(
                criteriaQuery.select(
                        criteriaQuery.from().count(),
                        get.sum()
                ).where(
                        criteriaQuery.from().get(Person.NAME).equal("Poliana")
                )
        );

This is actually working, I am already finishing implementing the MongoDB driver.

Are you sure you want me to go back to the old API ?
How can I expand the old API to do operations like these (count documents with predicates or do other aggregate operations) ?
Please double check the API (I pushed a small update today)

Otávio Santana · Answer 22 · Thu Apr 28 2022 23:34:21 GMT+0800 (China Standard Time)

@amoscatelli

The IDE is not to change the API, the API is perfect, only the implementation.
It would amazing if the CriteriaQuery<Person> returns the DocumentQuerysomehow.
The main goal is to have support for all the document types, not only for MongoDB.

Remember, the main point of the specification is to support more than one database.
If the goal is only to support MongoDB, initially, it is fine if it goes to extension and stays there for while.

amoscatelli · Answer 23 · Fri Apr 29 2022 22:09:20 GMT+0800 (China Standard Time)

@otaviojava I think you didn't understand

I am going to make an incomplete pull request for the communication layer implementation

We'll talk about this there

Otávio Santana · Answer 24 · Fri Apr 29 2022 22:21:27 GMT+0800 (China Standard Time)

@amoscatelli what do you think if we do a call?
It will make our life easier?
What do you think?

amoscatelli · Answer 25 · Fri Apr 29 2022 22:25:22 GMT+0800 (China Standard Time)

@otaviojava Sure it will make our life easier for this and the next functionalities (update criteria and delete criteria and I also have in mind something regarding polymoprhism).

Where would you prefer to make the call ? (telegram ? skype ? teams ?)
I am available from next monday

In the mean time here is the pull request
eclipse/jnosql-databases#178