Blob/object storage API?
salesportal opened this issue · comments
Hobby salesportal guy again
I am also utilising S3/local file system for test with a thin abstraction to store ad fields (as XML files) and photos.
Thus it uses only S3 and Elasticsearch, no need for other document store. Object store for this is very cheap and "good enough".
This could be yet another NoSQL technology that could be standardised. In its simplest form acts much like a filesystem (My abstaction interface is named IFileSystem) however there are some additional stuffs.
- multiple versions of objects, supported by S3, perhaps doable for Azure Blob Storage
- optimistic concurrency. Doable for Azure Blob Storage (HTTP If-Match), not supprted by S3 but could use DynamoDB to store hash on the side, thus achieve the same at slightly higher storage cost.
- indexing of object metadata for simple queries, seems to be supported by both S2 and Blob Storage.
Optimistic concurrency would be highly useful here as a lot of use cases where there are
- not much relations in the data
- one mostly has the main lookup keys (userID/ad ID) for direct lookup
- writes are infrequent
- search is done in separate index
- saving money on storage more important than convenience
Think of sales or ad portal, I can store ad text/fields and thumbnails as a zipped file cheaply (to get above S3 100kb per cost minimum, otherwise one pays 100kb cost for 5kb image) and then also store the fields in Elasticsarch for viewing so in gallery view, info is retrieved from document store for speed.
Hello, thank you for the feedback.
We already started that discussion on the last version.
https://github.com/eclipse/jnosql-diana-driver/issues/119
eclipse/jnosql-databases#120
Could you tell us what do you think about it?
This interface is already in the version 0.0.9:
https://mvnrepository.com/artifact/org.jnosql.diana/diana-driver-commons/0.0.9
Hi,
That might work, eg. one sees the queryable metadata of S3/Blob Storage as a document with document API and perhaps even simple select queries (S3 has those I guess) and the binary object is seen as an attachment to that document. So S3/Blob Storage as simple document DBs really.
However perhaps not all such blob storage services support querying metadata (eg google one https://cloud.google.com/storage/docs/how-to ?)
So then should one have an abstraction that is more filesystem like too? Such an API one could even use Dropbox as storage or a network filesystem or whatever.
My thinking is if app use case only needs to view it as file system, it can inject such a simple interface, if needs metadata it can inject interface to see it as a simple document store but that will narrow the number of available implementations.
Tradeoff against avoiding too many similar APIs though, blob stores are however sort of separate since they have the ability to have paths for the data and list items by path, which is something probably not in document DBs by default (though can be simulated by a path field on each document).
It has support; please, let me know if it works for you.