Some issues and Error format heterogeneity
sindbadz opened this issue · comments
Hi every one
First of all I would like to thank you for this great project!
I am starting to develop a mircoservice in node.js to deal with GC datastore and gstore will help me a lot.
After a week trying to build my requests (create, update, delete or get entities) I 'm facing some behaviours that seem unclear to me.
I will try to describe these points:
-Enity.save (update mode) vs Model.update
-Error issue with Joi and Model.update
-Result/Error heterogeneity
1/ Enity.save
(update mode) vs Model.update
I've tried both solutions to update my entities and discovered than when I use Model.update()
, only the new values where updated whereas using Entity.save(null, { method: 'update' })
, the all dataset stored in datastore was replace by the new one.
I might not perfectly understand the documentation, but I am not sure that the differences are explicitly described.
2/ Error issue with Joi and Model.update
I am using Joi to define my schemas and it works great in most of my tests. Neverthelss I found en little problem with Model.update error in the case where entity is not found (id not exising, or namespace not specified for example).
Indeed, that's what I get in these conditions with Entity.save(null, { method: 'update' })
:
{ Error: no entity to update: app: "..."
name_space: "..."
path <
Element {
type: "..."
id: ...
}
>
code: 5,
details: ...
} }
and that's what I obtain with Model.update()
(here with an unspecified namespace):
{ ValidationError: child "checkboxA1" fails because ["checkboxA1" is required]
isJoi: true,
name: 'ValidationError',
details:
[ { message: '"checkboxA1" is required',
path: [Array],
type: 'any.required',
context: [Object] } ],
_object:
{ [Symbol(KEY)]: Key { namespace: undefined, kind: '...', path: [Getter] } },
annotate: [Function] }
I have absolutely no validation issue regardless to my joi schema, this error appears only when entity is not found.
3/ Result/Error heterogeneity
I have notices some differences in errors and results format between the diffrent types of request I have done with Gstore.
Before using gstore, I was creating "hand-made" request to datastore.
In order to bring more explicit comparisons, I will also present errors/results obtained with hand-made requests.
a) update informations in results (create, update, delete)
- Create
-with hand-made request(datastore.insert(entity)
) :
[ { mutationResults:
[ { key:
{ path: [ { kind: '...', id: '...', idType: 'id' } ],
partitionId: { projectId: '...', namespaceId: '...' } },
version: '1558688427677000',
conflictDetected: false } ],
indexUpdates: 41 } ]
-with Entity.save({method: insert})
:
no informations
- Update
-with hand-made request (datastore.update(entity)
):
[ { mutationResults:
[ { key: null,
version: '1558686100928000',
conflictDetected: false } ],
indexUpdates: 4 } ]
-with Entity.save(null, { method: 'update' })
:
no informations
-withModel.update()
:
no informations
- Delete
-with hand-made request (datastore.delete()
):
{ mutationResults:
[ { key: null,
version: '1558688022947000',
conflictDetected: false } ],
indexUpdates: 41 }
-with Model.delete()
:
{ mutationResults:
[ { key: null,
version: '1558687176725000',
conflictDetected: false } ],
indexUpdates: 41,
key:
Key {
namespace: '...',
id: 5668779698683904,
kind: '...',
path: [Getter] },
success: true }
b) errors when no entity found (wrong id, missing namespace....)
- Update (wrong id, missing namespace)
-with hand-made request or with Entity.save(null, { method: 'update' })
:
{ Error: no entity to update: app: "..."
name_space: "..."
path <
Element {
type: "..."
id: ...
}
>
code: 5,
details: ...
} }
-with Model.update()
: previously exposed Bug
{ ValidationError: child "checkboxA1" fails because ["checkboxA1" is required]
isJoi: true,
name: 'ValidationError',
details:
[ { message: '"checkboxA1" is required',
path: [Array],
type: 'any.required',
context: [Object] } ],
_object:
{ [Symbol(KEY)]: Key { namespace: undefined, kind: '...', path: [Getter] } },
annotate: [Function] }
- Delete (wrong id, missing namespace)
-with hand-made request: no error, informations are in results
{ mutationResults:
[ { key: null,
version: '1558687176725000',
conflictDetected: false } ],
indexUpdates: 0 }
-with Model.delete()
: no error, informations are in results
{ mutationResults: [ { key: null, version: '1', conflictDetected: false } ],
indexUpdates: 0,
key:
Key { namespace: '...', id: ..., kind: '...', path: [Getter] },
success: false }
- Get
-with hand-made get (datastore.get(key)
) : no error, informations are in results (wrong id, missing namespace)
[ undefined ]
-with Model.get() (only one id): (wrong id, missing namespace)
{
GstoreError: MyKind { myid } not found
name: 'GstoreError',
code: 'ERR_ENTITY_NOT_FOUND'
}
-with hand-made get for array of ids (datastore.get(keys)
) : no error, informations are in results (wrong id, missing namespace)
result[0] = []
-with Model.get (array of ids): no error, informations are in results (wrong id)
[]
-with Model.get (array of ids): (missing namespace)
{ Error: Key path element must not be incomplete: [MyKind: ]
code: 3,
details: 'Key path element must not be incomplete: [MyKind: ]',
metadata: Metadata { internalRepr: Map {} },
note:
'Exception occurred in retry method that was not classified as transient' }
-with hand-made "get all entities" (datastore.runQuery(datastore .createQuery(namespace, kind))
: no error, informations are in results (missing namespace)
result[0] = []
-with "get all entities" Model.query(namespace).run(): no error, informations are in results (missing namespace)
{ entities: [] }
I have propably missed some infos in documentation, but if I don't, it could be good to have more homogeneous error formats
4/ Operator issues on queries filters
I have tried to get entities filtering regardless to the existence of a property but I had some issues.
What I have done:
myModel.query()
.filter('myPropetry', '!=', null)
.run()
or
myModel..findOne({ card: !null })
In both cases I get Error: a property filter must specify an operator
The errors appears also with the IN
operator.
Theses two operators do not seem available for Node.js (https://cloud.google.com/nodejs/docs/reference/datastore/1.4.x/Query).
I do not known if there is another way to make similar filters?
It might be interesting to list the available operators in the documentation.
I hope my post is clear enough.
Good week-end to you.
Hello,
Thanks for raising this and for your detailed analysis on the error/result. I am currently traveling so I can't answer each item separately as I'd like, but I will as soon as I am back (2 weeks aprox).
A quick note on the errors, there is indeed some differences like in the GET where the hand-made does not return anything when the entity is not found but where gstore does throw a 404 error in that case (there is, though, a global config setting to turn off that behaviour).
I thought that
{
GstoreError: MyKind { myid } not found
name: 'GstoreError',
code: 'ERR_ENTITY_NOT_FOUND'
}
was a better handled error for a entity that is not found error than... nothing. But that's opinated, I agree 😊
I will go through each of your use cases and comments. There is surely room for improvement.
As for the difference between the save()
with { method: 'update' }
vs Model.update()
. The Model.update()
does
- Model.get()
- Merge data
- Model.save()
In a single transaction, to ease updating only a few properties so you don't need to do the above manually.
The { method: 'update' }
means to only do the update if the entity already exists, but it will update (= replace) all the entity data.
Cheers
Hi,
Your wellcome :)
I agree that getting the :
{
GstoreError: MyKind { myid } not found
name: 'GstoreError',
code: 'ERR_ENTITY_NOT_FOUND'
}
is way better than getting nothing. I guess it could be even better if we get this error for all the different type of request when no entity is found.
Thanks a lot for your response.
Waiting for futher answers and solutions.
I have updated my first post adding point 4/ about query filter operators
Hi !
Is there any improvement about these points?
Hello, I am hoping to take back my OSS work next week. I am not sure how much time I will be able to dedicate but at least get something moving forward 😊
Sorry for so much delay...
1/ Enity.save (update mode) vs Model.update
Indeed both are different. The Entity.save(null, { method: 'update' })
maps to the google datastore update()
method. The Model.update()
is some sort of PATCH
to update an entity with a new prop, without the need to provide the complete entity data. It does for you, under the hood, a GET + SAVE inside a transaction.
[EDIT]: I updated the docs to make this clearer. Thanks for the input! 👍
2/ Error issue with Joi and Model.update
Could you open a separate issue and copy the content there? I will try to reproduce but if I can't it will be better to talk about it in a separate thread.
3/ Result/Error heterogeneity
By design, there was no intention to return the same error that the underlying google datatstore lib. So differences between the two ("hand-made" and gstore) are expected. I will have a look and decide if it makes sense to add a datastoreResponse
property to the gstore responses.
4/ Operator issues on queries filters
Indeed this 2 operators are not allowed. Here are the latest doc with the allowed operators:
https://googleapis.dev/nodejs/datastore/latest/Query.html#filter
I will link to them in the doc, thanks for the input!
Point 2 was indeed a bug and I opened a PR to fix it (#181). Thanks for raising it! 👍
When the PR will be merged it will automatically close this issue. Please re-open new issues for the other points you'd like to be addressed.
The fix will be available in the next major version (v7.0.0
)
i want to ask why populate always null
PublicCredit.list(options).populate('user') .then((credits) => { res.status(200).json(credits); console.log(credits, "tes") }) .catch(next);``PublicCredit.list(options).populate('user') .then((credits) => { res.status(200).json(credits); console.log(credits, "tes") }) .catch(next);