[BUG]: Add data no response when running image similarity search example application.
jaggerwang opened this issue · comments
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
I'm successfully running the image similarity search example application following this doc https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search/quick_deploy, but when I input /data to add images from PASCAL VOC image set, the request is pending for 55 minutes and then failed, and I see some warnning message in console.
Console log:
milvus-standalone | [2022/11/21 03:56:32.957 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-etcd | {"level":"info","ts":"2022-11-21T03:57:29.173Z","caller":"v3compactor/revision.go:86","msg":"starting auto revision compaction","revision":5700,"revision-compaction-retention":1000}
milvus-etcd | {"level":"info","ts":"2022-11-21T03:57:29.182Z","caller":"v3compactor/revision.go:94","msg":"completed auto revision compaction","revision":5700,"revision-compaction-retention":1000,"took":"8.498709ms"}
milvus-etcd | {"level":"info","ts":"2022-11-21T03:57:29.182Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":5700}
milvus-etcd | {"level":"info","ts":"2022-11-21T03:57:29.185Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":5700,"took":"907.583µs"}
milvus-standalone | [2022/11/21 03:57:30.962 +00:00] [INFO] [rocksmq_impl.go:228] ["Rocksmq stats"] [cache=4528] ["rockskv memtable "=2048000] ["rockskv table readers"=184] ["rockskv pinned"=0] ["store memtable "=2525328] ["store table readers"=1624] ["store pinned"=0] ["store l0 file num"=1] ["store l1 file num"=0] ["store l2 file num"=0] ["store l3 file num"=0] ["store l4 file num"=0]
milvus-standalone | [2022/11/21 03:57:32.078 +00:00] [WARN] [time_recorder.go:112] ["Datacoord haven't received tt for 2.000000 minutes"]
milvus-standalone | [2022/11/21 03:57:32.313 +00:00] [DEBUG] [root_coord.go:334] ["check flushed segments"]
milvus-standalone | [2022/11/21 03:57:32.937 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5931685499479156}"]
milvus-standalone | [2022/11/21 03:57:32.938 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-standalone | [2022/11/21 03:58:32.928 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5930096651602402}"]
milvus-standalone | [2022/11/21 03:58:32.929 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-standalone | [2022/11/21 03:59:32.078 +00:00] [WARN] [time_recorder.go:112] ["Datacoord haven't received tt for 2.000000 minutes"]
milvus-standalone | [2022/11/21 03:59:32.938 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5924952756601415}"]
milvus-standalone | [2022/11/21 03:59:32.943 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-standalone | [2022/11/21 04:00:32.920 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5919411649631239}"]
milvus-standalone | [2022/11/21 04:00:32.921 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-standalone | [2022/11/21 04:01:32.077 +00:00] [WARN] [time_recorder.go:112] ["Datacoord haven't received tt for 2.000000 minutes"]
milvus-standalone | [2022/11/21 04:01:32.924 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5940453953698986}"]
milvus-standalone | [2022/11/21 04:01:32.925 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-etcd | {"level":"info","ts":"2022-11-21T04:02:29.180Z","caller":"v3compactor/revision.go:86","msg":"starting auto revision compaction","revision":5998,"revision-compaction-retention":1000}
milvus-etcd | {"level":"info","ts":"2022-11-21T04:02:29.184Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":5998}
milvus-etcd | {"level":"info","ts":"2022-11-21T04:02:29.185Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":5998,"took":"426.958µs"}
milvus-etcd | {"level":"info","ts":"2022-11-21T04:02:29.185Z","caller":"v3compactor/revision.go:94","msg":"completed auto revision compaction","revision":5998,"revision-compaction-retention":1000,"took":"5.457292ms"}
milvus-standalone | [2022/11/21 04:02:30.962 +00:00] [INFO] [rocksmq_impl.go:228] ["Rocksmq stats"] [cache=4528] ["rockskv memtable "=2150400] ["rockskv table readers"=184] ["rockskv pinned"=0] ["store memtable "=2651856] ["store table readers"=1624] ["store pinned"=0] ["store l0 file num"=1] ["store l1 file num"=0] ["store l2 file num"=0] ["store l3 file num"=0] ["store l4 file num"=0]
milvus-standalone | [2022/11/21 04:02:32.922 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.592883550360023}"]
milvus-standalone | [2022/11/21 04:02:32.923 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-standalone | [2022/11/21 04:03:32.076 +00:00] [WARN] [time_recorder.go:112] ["Datacoord haven't received tt for 2.000000 minutes"]
milvus-standalone | [2022/11/21 04:03:32.910 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5920454331050358}"]
milvus-standalone | [2022/11/21 04:03:32.910 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-standalone | [2022/11/21 04:04:32.939 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5943125204191778}"]
milvus-standalone | [2022/11/21 04:04:32.941 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-standalone | [2022/11/21 04:05:32.127 +00:00] [WARN] [time_recorder.go:112] ["Datacoord haven't received tt for 2.000000 minutes"]
milvus-standalone | [2022/11/21 04:05:32.988 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.592104021870491}"]
milvus-standalone | [2022/11/21 04:05:32.989 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-standalone | [2022/11/21 04:06:32.961 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5933989328900448}"]
milvus-standalone | [2022/11/21 04:06:32.961 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
milvus-etcd | {"level":"info","ts":"2022-11-21T04:07:29.237Z","caller":"v3compactor/revision.go:86","msg":"starting auto revision compaction","revision":6298,"revision-compaction-retention":1000}
milvus-etcd | {"level":"info","ts":"2022-11-21T04:07:29.241Z","caller":"v3compactor/revision.go:94","msg":"completed auto revision compaction","revision":6298,"revision-compaction-retention":1000,"took":"4.058834ms"}
milvus-etcd | {"level":"info","ts":"2022-11-21T04:07:29.241Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":6298}
milvus-etcd | {"level":"info","ts":"2022-11-21T04:07:29.242Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":6298,"took":"599.833µs"}
milvus-standalone | [2022/11/21 04:07:31.016 +00:00] [INFO] [rocksmq_impl.go:228] ["Rocksmq stats"] [cache=4528] ["rockskv memtable "=2252552] ["rockskv table readers"=184] ["rockskv pinned"=0] ["store memtable "=2778056] ["store table readers"=1624] ["store pinned"=0] ["store l0 file num"=1] ["store l1 file num"=0] ["store l2 file num"=0] ["store l3 file num"=0] ["store l4 file num"=0]
milvus-standalone | [2022/11/21 04:07:32.127 +00:00] [WARN] [time_recorder.go:112] ["Datacoord haven't received tt for 2.000000 minutes"]
milvus-standalone | [2022/11/21 04:07:32.362 +00:00] [DEBUG] [root_coord.go:334] ["check flushed segments"]
milvus-standalone | [2022/11/21 04:07:32.987 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{\"11\":0.5933343859450517}"]
milvus-standalone | [2022/11/21 04:07:32.987 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[11]"] [availableNodeIDs="[11]"]
Expected Behavior
Add images successfully.
Steps To Reproduce
No response
Software version
Milvus: all using the latest images
Docker Desktop: 4.14.1
OS: MacOS 12.6Anything else?
No response