Block data retrieval error

Question

Block data retrieval error

hunte-star opened this issue 3 years ago · comments

Block data is not being pulled.
And once when the data was being pulled many block data were missing. Is there any particular parameter that might cause this issue?

hunte-star · Answer 1 · Wed Nov 24 2021 10:35:21 GMT+0800 (China Standard Time)

This is my substrate log. As you can see the block data retrieval is successful but is very rare.

2021/11/23 11:10:30 start watch filepath: ../configs
2021/11/23 11:10:30 - - - - - - - - - - - - - - -
2021/11/23 11:10:30 daemon started
WARN 11/23-11:10:32.076 /subscan/internal/service/service.go:43 Found unknown type Result<Null, ResponseV2Error>
INFO 11/23-11:10:35.070 /subscan/internal/service/substrate.go:170 Block num 1925 hash 0x397051af983017b39afc5470f24739300a1e769e786242517bfbf626c1f4f0b0
ERROR 11/23-11:10:35.624 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:35.624 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:35.625 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:35.626 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:35.628 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:35.631 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:35.632 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:35.634 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
INFO 11/23-11:10:35.857 /subscan/internal/service/substrate.go:170 Block num 1929 hash 0xe0d6ef2834bfcb994de909e356cbdd54368a2c0eaab2e8728fe8509f57eba214
ERROR 11/23-11:10:36.626 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:36.628 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:36.628 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:36.630 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:36.631 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:36.635 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:36.637 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:36.641 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:37.256 /subscan/internal/service/block.go:33 Recovering from panic in DecodeExtrinsic error is: Not find Extrinsic Lookup 0200, please check metadata info
ERROR 11/23-11:10:37.401 /subscan/internal/service/block.go:33 Recovering from panic in DecodeExtrinsic error is: Not find Extrinsic Lookup 0200, please check metadata info
ERROR 11/23-11:10:37.629 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:37.630 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:37.633 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:37.633 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:37.634 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:37.638 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:37.640 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:37.644 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
INFO 11/23-11:10:37.908 /subscan/internal/service/substrate.go:170 Block num 1950 hash 0xa5747ed638e838b2f03356ae6df14cf44b0970a59f4acbcb53a3b039b3f894e6
INFO 11/23-11:10:38.038 /subscan/internal/service/substrate.go:170 Block num 1960 hash 0x6e4c5345af7549df5bebef19c03cb12266f442a27977ef46060dfc83b61e5b19
ERROR 11/23-11:10:38.631 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:38.633 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:38.637 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:38.637 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:38.639 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:38.641 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:38.643 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:38.648 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:38.706 /subscan/internal/service/block.go:33 Recovering from panic in DecodeExtrinsic error is: Not find Extrinsic Lookup 0200, please check metadata info
ERROR 11/23-11:10:38.844 /subscan/internal/service/block.go:33 Recovering from panic in DecodeExtrinsic error is: Not find Extrinsic Lookup 0200, please check metadata info
ERROR 11/23-11:10:39.634 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:39.635 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:39.640 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:39.640 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:39.642 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:39.643 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:39.646 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:39.652 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:40.636 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:40.637 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:40.644 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:40.644 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:40.647 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:40.648 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:40.649 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:40.655 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.129 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: read tcp 172.18.0.5:53056->104.26.8.125:443: i/o timeout
ERROR 11/23-11:10:41.639 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.640 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.647 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.647 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.650 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.651 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.651 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.658 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:41.895 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.132 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.641 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.643 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.649 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.649 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.652 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.654 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.654 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.660 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:42.897 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:43.135 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:43.645 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:43.651 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:43.651 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:43.652 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected
ERROR 11/23-11:10:43.655 /subscan/internal/service/substrate.go:112 ChainGetBlockHash get error websocket send error: websocket send error: websocket: not connected

BoBin · Answer 2 · Wed Nov 24 2021 15:27:24 GMT+0800 (China Standard Time)

@hunte-star Looks like a network connection error, what endpoint are you using

hunte-star · Answer 3 · Wed Nov 24 2021 16:34:47 GMT+0800 (China Standard Time)

Currently the used end point is westend's wss://westend-rpc.polkadot.io
But this issue has been occurring even if i change the node to kusama or polkadot's socket.

BoBin · Answer 4 · Wed Nov 24 2021 16:48:05 GMT+0800 (China Standard Time)

@hunte-star Try use this wss://pub.elara.patract.io/polkadot

hunte-star · Answer 5 · Wed Nov 24 2021 17:07:30 GMT+0800 (China Standard Time)

No luck. Same problem. Plus i have used most of the endpoint from polkadot.js.org/apps the endpoint problem still exists.

@freehere107 now the data is being pulled correctly after i left it for some time. Thanks.
I have one more question the source json files inside the configs folder contains only 3 network but if i have to add other networks what do i need to do.

BoBin · Answer 6 · Thu Nov 25 2021 11:08:18 GMT+0800 (China Standard Time)

@hunte-star Add types.json like https://github.com/itering/scale.go/tree/master/network and change this env

https://github.com/itering/subscan-essentials/blob/6400a20d84b9b74afedb5c81d02044fc22202859/util/env.go#L14

hunte-star · Answer 7 · Thu Nov 25 2021 13:29:20 GMT+0800 (China Standard Time)

@freehere107 thanks.

hunte-star · Answer 8 · Thu Dec 30 2021 16:12:07 GMT+0800 (China Standard Time)

`2021/12/30 07:52:29 start watch filepath: ../configs

panic: Can not find chain metadata, please check network

panic: assignment to entry in nil map

goroutine 1 [running]:

github.com/itering/scale.go/types.regCustomKey(0xc00041ac40, 0x12, 0xc29d20, 0xc0005b43f0)

/go/pkg/mod/github.com/itering/scale.go@v1.1.42/types/customType.go:155 +0x4c9

github.com/itering/scale.go/types.RegCustomTypes(0xc000266e70)

/go/pkg/mod/github.com/itering/scale.go@v1.1.42/types/customType.go:115 +0x831

github.com/itering/substrate-api-rpc.RegCustomTypes(0xc0005be000, 0x1cdb, 0x1edb)

/go/pkg/mod/github.com/itering/substrate-api-rpc@v0.4.9/types.go:9 +0x4d

github.com/itering/subscan/internal/service.(*Service).initSubRuntimeLatest.func1(0xc000465300)

/subscan/internal/service/service.go:46 +0x5c

panic(0xb48960, 0xd6f9a0)

/usr/local/go/src/runtime/panic.go:969 +0x166

github.com/itering/subscan/internal/service.(*Service).initSubRuntimeLatest(0xc000465300)

/subscan/internal/service/service.go:67 +0x24b

github.com/itering/subscan/internal/service.New(0x10)

/subscan/internal/service/service.go:27 +0xa1

main.run()

/subscan/cmd/main.go:91 +0x34

main.setupApp.func1(0xc00035e160, 0x0, 0x0)

/subscan/cmd/main.go:38 +0x20

github.com/urfave/cli.HandleAction(0xb58a60, 0xca70c0, 0xc00035e160, 0xc000362480, 0x0)

/go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:490 +0xc8

github.com/urfave/cli.(*App).Run(0xc0000a8680, 0xc0001181c0, 0x1, 0x1, 0x0, 0x0)

/go/pkg/mod/github.com/urfave/cli@v1.20.0/app.go:264 +0x5dd

main.main()

/subscan/cmd/main.go:27 +0x71`

No matter which endpoint i use this is the error. I know it a endpoint connection problem but all the endpoint..