Search of documents in cloud-native (Google Kubernetes Cluster) fails
muruguru opened this issue · comments
We are trying to use Phalanx as a search engine and we see that Search of documents after inserting in cloud-native (GKE) as well as stand-along docker steps does not allow for the documents to be returned after a search POST call.
I am seeing the following error during initialization of Phalanx after few documents are inserted:
{"_level_":"error","_timestamp_":"2023-06-21T23:05:42.632-0700","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:144","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-4x76b7O1"}
While doing the search, I am seeing the following warning and no documents returned back:
{"_level_":"warn","_timestamp_":"2023-06-21T23:16:02.958-0700","_name_":"phalanx","_caller_":"server/index_service.go:1241","_message_":"no index readers are assigned","index_name":"example","shard_names":["shard-1TrQbp3v","shard-BALwDsbc","shard-M9UIob2J","shard-linNfk1K","shard-cgllS7mr","shard-Rjjugv2u","shard-4x76b7O1","shard-0AY1YYWM","shard-p5JJAniy","shard-ns5Tb3lf"]}
Our setup is as follows and we have tried both in GKE as well as standalone docker:
- Google Kubernetes Cluster - running etcd, minIO and phalanx as Pods
When running it in a docker mode, I see that the Search is actually crashing with SIGSEGV:
- Steps:
** run the docker command that is posted in the documentation above
** add the index with index_uri=file:///tmp/phalanx/metadata
** add few documents using curl -XPUT which I see those are added in the index and I could see that in the index metastore
** Try to search using curl POST call and I see SIGSEGV error
curl -X POST http://localhost:8000/v1/indexes/example/_search -d `{
"query": "text:example",
"boost": 1.0,
"start": 0,
"num": 10,
"sort_by": "-_score",
"fields": [
"id",
"text"
]
}`
api_1 | {"_level_":"info","_timestamp_":"2023-06-22T20:33:01.158Z","_name_":"phalanx","_caller_":"server/index_service.go:103","_message_":"shard metadata has been created","metastore_event":{"Type":3,"Index":"example","Shard":"shard-pWJ2XGkd"}} api_1 | {"_level_":"info","_timestamp_":"2023-06-22T20:33:01.158Z","_name_":"phalanx","_caller_":"server/index_service.go:198","_message_":"opening index writers"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.161Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-IGKNAfrL"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.161Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-IGKNAfrL"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.165Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-m65GeEjC"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.165Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-m65GeEjC"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.168Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-Bx05hbMk"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.168Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-Bx05hbMk"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.171Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-yFbRK33L"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.171Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-yFbRK33L"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.174Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-Hp3BVksK"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.174Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-Hp3BVksK"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.177Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-DSwnLR02"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.177Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-DSwnLR02"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:33:01.180Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:129","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-ecaa880o"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:33:01.180Z","_name_":"phalanx","_caller_":"server/index_service.go:301","_message_":"error opening index: unable to find a usable snapshot","index_name":"example","shard_name":"shard-ecaa880o"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-m65GeEjC"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-m65GeEjC"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-Bx05hbMk"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-Bx05hbMk"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-IGKNAfrL"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-IGKNAfrL"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-DSwnLR02"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-DSwnLR02"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-ecaa880o"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-ecaa880o"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-yFbRK33L"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-yFbRK33L"} api_1 | {"_level_":"error","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx.manager.reader","_caller_":"index/reader.go:162","_message_":"shard does not exist","index_name":"example","shard_name":"shard-Hp3BVksK"} api_1 | {"_level_":"warn","_timestamp_":"2023-06-22T20:34:23.968Z","_name_":"phalanx","_caller_":"server/index_service.go:1088","_message_":"shard does not exist","index_name":"example","shard_name":"shard-Hp3BVksK"} api_1 | panic: runtime error: invalid memory address or nil pointer dereference api_1 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x122640b] api_1 | api_1 | goroutine 75 [running]: api_1 | github.com/mosuka/phalanx/server.(*IndexService).Search.func1() api_1 | /go/src/github.com/mosuka/phalanx/server/index_service.go:1108 +0x120b api_1 | golang.org/x/sync/errgroup.(*Group).Go.func1() api_1 | /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67 api_1 | created by golang.org/x/sync/errgroup.(*Group).Go api_1 | /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92 phalanx_api_1 exited with code 2
Any help is appreciated.