Giters

apache / incubator-horaedb-meta

Meta service of HoraeDB cluster.

Home Page:https://horaedb.apache.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

CeresMeta follower node did not forward create/drop table request to leader

ZuLiangWang opened this issue · comments

commented

Describe this problem
When CeresDB is started in dynamic cluster mode, CeresMeta node will not forward create/drop table request after receiving it, but directly process it. Because follower node does not have complete memory data, it will get error like this:

io.ceresdb.http.errors.ManagementException: Execute sql [CREATE TABLE MY_FIRST_TABLE31(ts TIMESTAMP NOT NULL,c1 STRING TAG NOT NULL,c2 STRING TAG NOT NULL,c3 DOUBLE NULL,c4 STRING NULL,c5 INT64 NULL,c6 FLOAT NULL,c7 INT32 NULL,c8 INT16 NULL,c9 INT8 NULL,c10 BOOLEAN NULL,c11 UINT64 NULL,c12 UINT32 NULL,c13 UINT16 NULL,c14 UINT8 NULL,c15 TIMESTAMP NULL,c16 VARBINARY NULL,TIMESTAMP KEY(ts)) ENGINE=Analytic] error from server 127.0.0.1:5440, err_code=500, err_msg=Internal Server Error, detail_msg={"code":500,"message":"Failed to handle request, err:Failed to execute interpreter, query:CREATE TABLE MY_FIRST_TABLE31(ts TIMESTAMP NOT NULL,c1 STRING TAG NOT NULL,c2 STRING TAG NOT NULL,c3 DOUBLE NULL,c4 STRING NULL,c5 INT64 NULL,c6 FLOAT NULL,c7 INT32 NULL,c8 INT16 NULL,c9 INT8 NULL,c10 BOOLEAN NULL,c11 UINT64 NULL,c12 UINT32 NULL,c13 UINT16 NULL,c14 UINT8 NULL,c15 TIMESTAMP NULL,c16 VARBINARY NULL,TIMESTAMP KEY(ts)) ENGINE=Analytic, err:Failed to execute create table, err:Failed to create table by table manipulator, err:Failed to create table, msg:failed to create table by meta client, req:CreateTableRequest { schema_name: \"public\", name: \"MY_FIRST_TABLE31\", encoded_schema: [0, 10, 8, 10, 2, 116, 115, 16, 1, 32, 1, 10, 10, 10, 4, 116, 115, 105, 100, 16, 5, 32, 2, 10, 10, 10, 2, 99, 49, 16, 4, 32, 3, 40, 1, 10, 11, 10, 3, 99, 49, 48, 16, 14, 24, 1, 32, 4, 10, 11, 10, 3, 99, 49, 49, 16, 5, 24, 1, 32, 5, 10, 11, 10, 3, 99, 49, 50, 16, 11, 24, 1, 32, 6, 10, 11, 10, 3, 99, 49, 51, 16, 12, 24, 1, 32, 7, 10, 11, 10, 3, 99, 49, 52, 16, 13, 24, 1, 32, 8, 10, 11, 10, 3, 99, 49, 53, 16, 1, 24, 1, 32, 9, 10, 11, 10, 3, 99, 49, 54, 16, 3, 24, 1, 32, 10, 10, 10, 10, 2, 99, 50, 16, 4, 32, 11, 40, 1, 10, 10, 10, 2, 99, 51, 16, 2, 24, 1, 32, 12, 10, 10, 10, 2, 99, 52, 16, 4, 24, 1, 32, 13, 10, 10, 10, 2, 99, 53, 16, 7, 24, 1, 32, 14, 10, 10, 10, 2, 99, 54, 16, 6, 24, 1, 32, 15, 10, 10, 10, 2, 99, 55, 16, 8, 24, 1, 32, 16, 10, 10, 10, 2, 99, 56, 16, 9, 24, 1, 32, 17, 10, 10, 10, 2, 99, 57, 16, 10, 24, 1, 32, 18, 16, 1, 24, 2, 40, 1], engine: \"Analytic\", create_if_not_exist: false, options: {} }, err:Bad response, resp code:404, msg:create table(#404)cluster not found, cause:<nil>."}

Steps to reproduce

  1. Set the startup parameter meta of CeresDB cluster meta_addr to CeresMeta follower node address.
  2. Start CeresDB in cluster mode.
  3. Send create table request by client.

CeresDB error info:

2022-11-04 14:17:06.043 ERRO [server/src/http.rs:140] Http service Failed to handle sql, err:Failed to execute interpreter, query:CREATE TABLE MY_FIRST_TABLE31(ts TIMESTAMP NOT NULL,c1 STRING TAG NOT NULL,c2 STRING TAG NOT NULL,c3 DOUBLE NULL,c4 STRING NULL,c5 INT64 NULL,c6 FLOAT NULL,c7 INT32 NULL,c8 INT16 NULL,c9 INT8 NULL,c10 BOOLEAN NULL,c11 UINT64 NULL,c12 UINT32 NULL,c13 UINT16 NULL,c14 UINT8 NULL,c15 TIMESTAMP NULL,c16 VARBINARY NULL,TIMESTAMP KEY(ts)) ENGINE=Analytic, err:Failed to execute create table, err:Failed to create table by table manipulator, err:Failed to create table, msg:failed to create table by meta client, req:CreateTableRequest { schema_name: "public", name: "MY_FIRST_TABLE31", encoded_schema: [0, 10, 8, 10, 2, 116, 115, 16, 1, 32, 1, 10, 10, 10, 4, 116, 115, 105, 100, 16, 5, 32, 2, 10, 10, 10, 2, 99, 49, 16, 4, 32, 3, 40, 1, 10, 11, 10, 3, 99, 49, 48, 16, 14, 24, 1, 32, 4, 10, 11, 10, 3, 99, 49, 49, 16, 5, 24, 1, 32, 5, 10, 11, 10, 3, 99, 49, 50, 16, 11, 24, 1, 32, 6, 10, 11, 10, 3, 99, 49, 51, 16, 12, 24, 1, 32, 7, 10, 11, 10, 3, 99, 49, 52, 16, 13, 24, 1, 32, 8, 10, 11, 10, 3, 99, 49, 53, 16, 1, 24, 1, 32, 9, 10, 11, 10, 3, 99, 49, 54, 16, 3, 24, 1, 32, 10, 10, 10, 10, 2, 99, 50, 16, 4, 32, 11, 40, 1, 10, 10, 10, 2, 99, 51, 16, 2, 24, 1, 32, 12, 10, 10, 10, 2, 99, 52, 16, 4, 24, 1, 32, 13, 10, 10, 10, 2, 99, 53, 16, 7, 24, 1, 32, 14, 10, 10, 10, 2, 99, 54, 16, 6, 24, 1, 32, 15, 10, 10, 10, 2, 99, 55, 16, 8, 24, 1, 32, 16, 10, 10, 10, 2, 99, 56, 16, 9, 24, 1, 32, 17, 10, 10, 10, 2, 99, 57, 16, 10, 24, 1, 32, 18, 16, 1, 24, 2, 40, 1], engine: "Analytic", create_if_not_exist: false, options: {} }, err:Bad response, resp code:404, msg:create table(#404)cluster not found, cause:<nil>.
Backtrace:
  0 backtrace::backtrace::libunwind::trace::h27a550175f4658ea
    /Users/zulliangwang/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.66/src/backtrace/libunwind.rs:93
    backtrace::backtrace::trace_unsynchronized::hc0bc2725eecb7c13
    /Users/zulliangwang/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.66/src/backtrace/mod.rs:66
  1 backtrace::backtrace::trace::hfe93c015b76a8a27
    /Users/zulliangwang/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.66/src/backtrace/mod.rs:53
  2 backtrace::capture::Backtrace::create::h2dd73475d25bc41c

CeresMeta error info:

2022-11-04T14:17:05.373+0800    error   grpc/service.go:121     fail to create table    {"error": "(#404)cluster not found, cause:<nil>"}
github.com/CeresDB/ceresmeta/server/service/grpc.(*Service).CreateTable
        /Users/zulliangwang/code/ceres/ceresmeta/server/service/grpc/service.go:121
github.com/CeresDB/ceresdbproto/pkg/metaservicepb._CeresmetaRpcService_CreateTable_Handler.func1
        /Users/zulliangwang/go/pkg/mod/github.com/!ceres!d!b/ceresdbproto@v0.0.0-20221019023918-29cb0c6fba76/pkg/metaservicepb/meta_service_grpc.pb.go:206
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1
        /Users/zulliangwang/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /Users/zulliangwang/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.newUnaryInterceptor.func1
        /Users/zulliangwang/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/api/v3rpc/interceptor.go:71
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /Users/zulliangwang/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1
        /Users/zulliangwang/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.4/etcdserver/api/v3rpc/interceptor.go:78
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /Users/zulliangwang/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /Users/zulliangwang/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34
github.com/CeresDB/ceresdbproto/pkg/metaservicepb._CeresmetaRpcService_CreateTable_Handler
        /Users/zulliangwang/go/pkg/mod/github.com/!ceres!d!b/ceresdbproto@v0.0.0-20221019023918-29cb0c6fba76/pkg/metaservicepb/meta_service_grpc.pb.go:208
google.golang.org/grpc.(*Server).processUnaryRPC
        /Users/zulliangwang/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1283
google.golang.org/grpc.(*Server).handleStream
        /Users/zulliangwang/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1620
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /Users/zulliangwang/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:922

Expected behavior
The create/drop table request sent by the client can be processed normally

Additional Information