Add annotations to support Redis's RediSearch ephemeral indexes
fcerbell opened this issue · comments
Idea :
- Annotation with ephemeral options (TTL, delete docs, ...)
- Index lazy created if not already exists at FindBy call.
@fcerbell I've been thinking of how to do this in a Spring-way fashion. I'm happy to get on a call to discuss the idea.
@fcerbell the documented use case https://redis.com/blog/the-case-for-ephemeral-search/ relies on adding documents manually to the index, which is deprecated I believe. The example relies on data duplication mostly:
At login, the index would be populated with FT.ADD from the other data source. Nothing special needed here—Search and Query will take care of the document and keys as temporary with no other syntax. Adding the documents will be quick—in the low-single-digit-milliseconds range for most documents. This doesn’t have to be done synchronously, so as a user initially browses the site, the purchase history can be loaded in the background.
One general note about Search and Query that should be restated, especially in this multi-index context: all document names should be unique across all indexes to prevent key contention at the hash level. Finally, in some circumstances you can save space by using the NOSAVE option on FT.ADD. This will not store the document, but rather just index it, providing you with the document ID only on FT.SEARCH, though this does complicate the result retrieval process.
So big assumption now is that if you have an ephemeral index you also have ephemeral data. What we can do currently is:
- Expose a method in the indexer to allow the creation of temporary/ephemeral indices given a class/entity (annotated with @indexed and @searchble)
- A way to mark those annotated entities with something to prevent a regular index to be created on startup if non-existing
- A way to query temporary indices. This will likely be using EntityStream API at first
@fcerbell some issues with Ephemeral indices at the moment...
Posed the question:
Ephemeral Indices:
* I'm implementing a couple of use cases for Ephemeral Indices in Redis OM Spring, first use case is "User already has a large collection of documents/hashes and wants a temporary index narrowed down to a specific user/account"
* Is there a way to prevent the automated dropping of the index from deleting the records associated with the index?
Current answer:
Unfortunately if you using the TTL in this case, it will mimic the FT.DROPINDEX…..DD after x seconds of inactivity. Eventually we should follow the same behaviour here also
. Adding a ticket for review it. A workaround will be setting this logic in the application using FT.CREATE + FT.DROPINDEX (without DD)