Add support for UUIDs and ULIDs as TAG indexable field

Question

Add support for UUIDs and ULIDs as TAG indexable field

radhakrishna67 opened this issue 10 months ago · comments

radhakrishna67 commented 10 months ago

I have tried with json type but getting indexing issues when adding second record to the index.

Artem Artemyev · Answer 1 · Sat Nov 18 2023 04:33:45 GMT+0800 (China Standard Time)

Hello! could you explain a little more? if possible, sample code

radhakrishna67 · Answer 2 · Mon Nov 20 2023 17:36:39 GMT+0800 (China Standard Time)

@bsbodden will refer HPE slack channel for full details along with entities.

Brian Sam-Bodden · Answer 3 · Wed Nov 22 2023 22:06:33 GMT+0800 (China Standard Time)

@bsbodden will refer HPE slack channel for full details along with entities.

@radhakrishna67 what we need to help is a "reproducer" - a full working example showing the issue using simplified code, just the minimum that can be use to craft a test and have others like @justTimTim above be able to help

radhakrishna67 · Answer 4 · Mon Dec 04 2023 23:52:03 GMT+0800 (China Standard Time)

@bsbodden ,
here is the sample application to reproduce this issue,
https://github.com/radhakrishna67/demo.git

Steps to reproduce:
Call http://localhost:8081/chats URL using GET method

Call http://localhost:8081/chats/aec5b048-92ae-11ee-bb93-00ff19b1c127 (chatId created in the database from the step one) using POST method with following JSON body:
{ "messageId": "8eb44965-7a31-11ee-bfdd-005056b843d2", "query": "where are you", "answer": "As an AI assistant, I don't have a physical location. I exist to provide information about company, its products, and service offerings. How can I assist you today?", "isSaved": true, "isRegenerate": true, "timestamp": "2023-11-03T10:12:56.123" }
Call the first GET method to see the response.
Call the send step to add second message.
We can see entire JSON record is not returned when called using GET method or using cli command
ft.search "com.example.demo.model.UserChatIdx" "@userid: {00uqgwatgxIFc86w0357}"
J

Brian Sam-Bodden · Answer 5 · Wed Dec 06 2023 10:21:45 GMT+0800 (China Standard Time)

@radhakrishna67 can you post your application-local.properties - the app does not run as is, I tried putting values in application.properties to no avail

radhakrishna67 · Answer 6 · Wed Dec 06 2023 23:55:27 GMT+0800 (China Standard Time)

@bsbodden , can you please try with the following properties,

Redis Configuration

spring.redis.database=0
spring.redis.host=host_name
spring.redis.port=port_number
spring.redis.password=db_password
spring.redis.timeout=60000

ghostwriter.chat.share.url=localhost

Brian Sam-Bodden · Answer 7 · Thu Dec 07 2023 04:30:00 GMT+0800 (China Standard Time)

@radhakrishna67 This works fine for me (notice the case on the @userId, you had @userid above:

127.0.0.1:6379> ft.search "com.example.demo.model.UserChatIdx" "@userId: {00uqgwatgxIFc86w0357}"
1) (integer) 1
2) "UserChat:00uqgwatgxIFc86w0357"
3) 1) "$"
   2) "{\"userId\":\"00uqgwatgxIFc86w0357\",\"chats\":[{\"chatId\":\"539ff9c4-9458-11ee-84b0-acde48001122\",\"title\":\"2023-12-06 09:55:57\",\"messages\":[{\"messageId\":\"8eb44965-7a31-11ee-bfdd-005056b843d2\",\"query\":\"where are you\",\"answer\":\"As an AI assistant, I don't have a physical location. I exist to provide information about company, its products, and service offerings. How can I assist you today?\",\"isSaved\":true,\"isRegenerate\":true,\"timestamp\":1699031576123}],\"timestamp\":1701881757609}],\"savedMessages\":[{\"chatId\":\"539ff9c4-9458-11ee-84b0-acde48001122\",\"messageId\":\"8eb44965-7a31-11ee-bfdd-005056b843d2\",\"query\":\"where are you\",\"answer\":\"As an AI assistant, I don't have a physical location. I exist to provide information about company, its products, and service offerings. How can I assist you today?\",\"isSaved\":true,\"isRegenerate\":true,\"timestamp\":1699031576123}]}"

radhakrishna67 · Answer 8 · Thu Dec 07 2023 23:26:38 GMT+0800 (China Standard Time)

@bsbodden , yes noticed it. can you please confirm that you could reproduce the using after following the steps given with Call http://localhost:8081/chats URL using GET method

Brian Sam-Bodden · Answer 9 · Fri Dec 08 2023 00:05:57 GMT+0800 (China Standard Time)

@radhakrishna67 I seem to get all the messages using the GET request. I still do not understand what the error that you're getting is or if the below is the expected response. Clarify please.

radhakrishna67 · Answer 10 · Fri Dec 08 2023 13:03:00 GMT+0800 (China Standard Time)

@bsbodden ,
It works totally fine when there is only one record inserted.
Please insert second record and read all of them. The index problem occurs when second record is inserted.

radhakrishna67 · Answer 11 · Wed Dec 13 2023 14:22:02 GMT+0800 (China Standard Time)

@bsbodden , is there any update on this issue?

Brian Sam-Bodden · Answer 12 · Wed Dec 20 2023 10:35:53 GMT+0800 (China Standard Time)

{ "messageId": "8eb44965-7a31-11ee-bfdd-005056b843d2", "query": "where are you", "answer": "As an AI assistant, I don't have a physical location. I exist to provide information about company, its products, and service offerings. How can I assist you today?", "isSaved": true, "isRegenerate": true, "timestamp": "2023-11-03T10:12:56.123" }

The second post does not seem to add an entry to the JSON object
The associated JSON.SET sends the exact same JSON
Therefore the JSON.GET will return the same

This does not seem to be an indexing issue, but an issue in the saveOrUpdateChatMessage logic. Below is the sequence of Redis commands when executing your test:

127.0.0.1:6379> MONITOR
OK
1703038293.613662 [0 172.20.0.1:60588] "FT.CREATE" "com.example.demo.model.ChatShareIdx" "ON" "JSON" "PREFIX" "1" "ChatShare" "LANGUAGE" "english" "SCHEMA" "$.userId" "AS" "userId" "TAG" "SEPARATOR" "|" "$.chatId" "AS" "chatId" "TAG" "SEPARATOR" "|" "$.timestamp" "AS" "timestamp" "NUMERIC" "$.shareId" "AS" "shareId" "TAG" "SEPARATOR" "|"
1703038293.620609 [0 172.20.0.1:60588] "FT.CREATE" "com.example.demo.model.UserChatIdx" "ON" "JSON" "PREFIX" "1" "UserChat" "LANGUAGE" "english" "SCHEMA" "$.savedMessages[0:].chatId" "AS" "savedMessages_chatId" "TAG" "SEPARATOR" "|" "$.savedMessages[0:].query" "AS" "savedMessages_query" "TAG" "SEPARATOR" "|" "$.savedMessages[0:].isSaved" "AS" "savedMessages_isSaved" "TAG" "SEPARATOR" "|" "$.savedMessages[0:].isRegenerate" "AS" "savedMessages_isRegenerate" "TAG" "SEPARATOR" "|" "$.savedMessages[0:].timestamp" "AS" "savedMessages_timestamp" "NUMERIC" "$.userId" "AS" "userId" "TAG" "SEPARATOR" "|"

1703038309.700176 [0 172.20.0.1:60588] "JSON.GET" "UserChat:00uqgwatgxIFc86w0357" "."
1703038309.714736 [0 172.20.0.1:60588] "FT.SEARCH" "com.example.demo.model.UserChatIdx" "( @userId:{00uqgwatgxIFc86w0357})" "LIMIT" "0" "1" "DIALECT" "1"
1703038309.778236 [0 172.20.0.1:38288] "EXISTS" "UserChat:00uqgwatgxIFc86w0357"
1703038309.786306 [0 172.20.0.1:60588] "JSON.SET" "UserChat:00uqgwatgxIFc86w0357" "." "{\"userId\":\"00uqgwatgxIFc86w0357\",\"chats\":[{\"chatId\":\"22663a4f-9edd-11ee-9af4-acde48001122\",\"title\":\"2023-12-19 19:11:49\",\"messages\":[],\"timestamp\":1703038309716}],\"savedMessages\":[]}"
1703038322.801137 [0 172.20.0.1:38288] "PING"
1703038322.820132 [0 172.20.0.1:60588] "PING"

1703038329.407392 [0 172.20.0.1:60588] "JSON.GET" "UserChat:00uqgwatgxIFc86w0357" "."
1703038329.413659 [0 172.20.0.1:38288] "EXISTS" "UserChat:00uqgwatgxIFc86w0357"

1703038329.415918 [0 172.20.0.1:60588] "JSON.SET" "UserChat:00uqgwatgxIFc86w0357" "." "{\"userId\":\"00uqgwatgxIFc86w0357\",\"chats\":[{\"chatId\":\"22663a4f-9edd-11ee-9af4-acde48001122\",\"title\":\"2023-12-19 19:11:49\",\"messages\":[{\"messageId\":\"8eb44965-7a31-11ee-bfdd-005056b843d2\",\"query\":\"where are you\",\"answer\":\"As an AI assistant, I don\\u0027t have a physical location. I exist to provide information about company, its products, and service offerings. How can I assist you today?\",\"isSaved\":true,\"isRegenerate\":true,\"timestamp\":1699031576123}],\"timestamp\":1703038309716}],\"savedMessages\":[{\"chatId\":\"22663a4f-9edd-11ee-9af4-acde48001122\",\"messageId\":\"8eb44965-7a31-11ee-bfdd-005056b843d2\",\"query\":\"where are you\",\"answer\":\"As an AI assistant, I don\\u0027t have a physical location. I exist to provide information about company, its products, and service offerings. How can I assist you today?\",\"isSaved\":true,\"isRegenerate\":true,\"timestamp\":1699031576123}]}"

1703038352.802417 [0 172.20.0.1:38288] "PING"
1703038352.825043 [0 172.20.0.1:60588] "PING"

1703038361.444084 [0 172.20.0.1:60588] "JSON.GET" "UserChat:00uqgwatgxIFc86w0357" "."
1703038361.447202 [0 172.20.0.1:60588] "FT.SEARCH" "com.example.demo.model.UserChatIdx" "( @userId:{00uqgwatgxIFc86w0357})" "LIMIT" "0" "1" "DIALECT" "1"

1703038364.767201 [0 172.20.0.1:60588] "JSON.GET" "UserChat:00uqgwatgxIFc86w0357" "."
1703038364.769194 [0 172.20.0.1:38288] "EXISTS" "UserChat:00uqgwatgxIFc86w0357"
1703038364.770811 [0 172.20.0.1:60588] "JSON.SET" "UserChat:00uqgwatgxIFc86w0357" "." "{\"userId\":\"00uqgwatgxIFc86w0357\",\"chats\":[{\"chatId\":\"22663a4f-9edd-11ee-9af4-acde48001122\",\"title\":\"2023-12-19 19:11:49\",\"messages\":[{\"messageId\":\"8eb44965-7a31-11ee-bfdd-005056b843d2\",\"query\":\"where are you\",\"answer\":\"As an AI assistant, I don\\u0027t have a physical location. I exist to provide information about company, its products, and service offerings. How can I assist you today?\",\"isSaved\":true,\"isRegenerate\":true,\"timestamp\":1699031576123}],\"timestamp\":1703038309716}],\"savedMessages\":[{\"chatId\":\"22663a4f-9edd-11ee-9af4-acde48001122\",\"messageId\":\"8eb44965-7a31-11ee-bfdd-005056b843d2\",\"query\":\"where are you\",\"answer\":\"As an AI assistant, I don\\u0027t have a physical location. I exist to provide information about company, its products, and service offerings. How can I assist you today?\",\"isSaved\":true,\"isRegenerate\":true,\"timestamp\":1699031576123}]}"

radhakrishna67 · Answer 13 · Wed Jan 10 2024 17:40:09 GMT+0800 (China Standard Time)

@bsbodden , The initial issue is the same that is resulting in all these issues.
Issue: JSON subdocument having list is not getting creating indexing definition using om spring.

`
@DaTa
@tostring
@builder
@document("UserChat")
@requiredargsconstructor(staticName = "of")
@AllArgsConstructor(access = AccessLevel.PROTECTED)
public class UserChat {

@Id
private String userId;
@Indexed
private List<Chat> chats;
@Indexed
private List<SavedMessage> savedMessages;

}
`

`
@DaTa
@requiredargsconstructor(staticName = "of")
public class Chat {
@indexed
@notempty(message ="chatId field value should not be null or empty")
private UUID chatId;

@Indexed
@NotEmpty(message ="chatTitle field value should not be null or empty")
private String title;

private List<Message> messages;

private LocalDateTime timestamp;

}
`

@Data @RequiredArgsConstructor(staticName = "of") public class SavedMessage { @Indexed private String chatId; @Indexed private UUID messageId; @Indexed private String query; @Indexed private String answer; private Boolean isSaved; private Boolean isRegenerate; @Indexed private LocalDateTime timestamp; }

@Data @RequiredArgsConstructor(staticName = "of") public class Message { @Indexed private UUID messageId; private String query; private String answer; private Boolean isSaved; private Boolean isRegenerate; private LocalDateTime timestamp; }
2024-01-10T14:59:59.182+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Creating index for UserChat annotated Entity... 2024-01-10T14:59:59.183+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Found @JSON annotated class: com.hpe.ug.session.model.UserChat 2024-01-10T14:59:59.183+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Found @Indexed annotation on field of type: interface java.util.List 2024-01-10T14:59:59.183+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Creating nested relationships: chats -> title 2024-01-10T14:59:59.183+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Found @Indexed annotation on field of type: interface java.util.List 2024-01-10T14:59:59.183+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Creating nested relationships: savedMessages -> chatId 2024-01-10T14:59:59.184+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Creating nested relationships: savedMessages -> query 2024-01-10T14:59:59.185+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Creating nested relationships: savedMessages -> answer 2024-01-10T14:59:59.185+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Creating nested relationships: savedMessages -> timestamp 2024-01-10T14:59:59.185+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Using entity prefix 'UserChat:' as keyspace for type : class com.hpe.ug.session.model.UserChat 2024-01-10T14:59:59.479+05:30 INFO 25616 --- [ restartedMain] com.redis.om.spring.RediSearchIndexer : Found 0 @RedisHash annotated Beans...

"com.hpe.ug.session.model.UserChatIdx" "ON" "JSON" "PREFIX" "1" "UserChat" "LANGUAGE" "english" "SCHEMA" "$.chats[0:].title" "AS" "chats_title" "TAG" "SEPARATOR" "|" "$.savedMessages[0:].chatId" "AS" "savedMessages_chatId" "TAG" "SEPARATOR" "|" "$.savedMessages[0:].query" "AS" "savedMessages_query" "TAG" "SEPARATOR" "|" "$.savedMessages[0:].answer" "AS" "savedMessages_answer" "TAG" "SEPARATOR" "|" "$.savedMessages[0:].timestamp" "AS" "savedMessages_timestamp" "NUMERIC" "$.userId" "AS" "userId" "TAG" "SEPARATOR" "|"

Index creating is excluding chatId, messages and timestamp from Chat class.