Performance degradation on a long running network
askolesov opened this issue · comments
Description:
We have a demo network of 4 nodes running in Aws. At the start, the network performs well, blocks are committed according to configured timeouts. After a couple of weeks, performance starts degrading. Intervals between blocks become about 30 seconds. Height of one of the nodes becomes much less than the height of others.
There is still a lot of RAM (>3 GB) and SSD (>10 GB) free space. Size of DB folder is about 4 GB.
The most interesting part is that restarting of all node processes helps to fix the issue. Not sure if the laggard node is able to catch up others, but other nodes start working as at the beginning.
It would be great to know how to fix this issue or at least where to look at and how to efficiently debug such cases.
Thanks!
Configuration:
Network size: 4 nodes
Node OS: Ubuntu 18.04
Exonum version: 1.0.0
Consensus configuration:
[consensus]
first_round_timeout = 3000
max_message_len = 1048576
max_propose_timeout = 2000
min_propose_timeout = 1000
peers_timeout = 10000
propose_timeout_threshold = 50
status_timeout = 5000
txs_block_limit = 1000Steps to reproduce:
- Strart a network of 4 nodes in the same region in AWS
- Wait for a cople of weeks
Result:
Network performance is degraded. Intervals between blocks become about 30 seconds.
Expected result:
Network performance is not degraded.
Hello. Are sending any transactions ? How many transactions included into a block ?
Hello. Are sending any transactions ? How many transactions included into a block ?
Few dozen at the beginning and a couple of them later. Less than 100 in total.
Is the time service included into a node ?
No, we have only one instance of our artefact. Our service is written in Rust.
Hm. With such tiny load it is a really weird behaviour. Could you please attach some logs from laggard node?
Unfortunately, I only have access to the network in the evening but I have some old logs. They are similar for all of 4 nodes. I think log level was info. Here is example:
[2020-08-01T06:04:08.003899853Z ERROR exonum_node::consensus] Received incorrect block BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194061), tx_count: 0, prev_hash: Hash(6c27b66f...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [13, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [2, 0]}) } }, precommits: [[10, 96, 18, 94, 8, 1, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 241, 238, 158, 182, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 214, 203, 195, 9, 83, 214, 126, 250, 63, 194, 111, 153, 142, 215, 41, 135, 7, 183, 234, 78, 227, 113, 150, 104, 85, 17, 170, 81, 155, 202, 18, 75, 208, 175, 220, 206, 191, 142, 230, 218, 240, 152, 157, 145, 130, 15, 26, 122, 5, 6, 148, 200, 61, 170, 76, 49, 150, 138, 199, 189, 212, 86, 50, 4], [10, 96, 18, 94, 8, 2, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 141, 228, 246, 176, 2, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 204, 115, 84, 55, 124, 16, 82, 4, 99, 139, 20, 13, 169, 244, 149, 209, 36, 62, 12, 5, 44, 1, 114, 235, 96, 182, 149, 225, 32, 22, 140, 158, 230, 140, 190, 203, 90, 95, 98, 216, 54, 60, 141, 178, 59, 67, 114, 40, 251, 234, 221, 136, 206, 249, 112, 72, 136, 220, 241, 217, 197, 147, 255, 4], [10, 96, 18, 94, 8, 3, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 217, 155, 167, 233, 2, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 105, 240, 121, 221, 0, 44, 111, 169, 143, 156, 198, 41, 127, 171, 129, 117, 145, 76, 60, 160, 73, 189, 93, 231, 80, 70, 254, 237, 169, 53, 82, 114, 87, 55, 166, 70, 151, 80, 155, 83, 44, 15, 74, 206, 147, 52, 246, 192, 15, 6, 71, 224, 241, 107, 166, 237, 0, 6, 187, 58, 96, 96, 107, 14]], transactions: [] }: Received block has inappropriate height or epoch, msg=Verified { raw: SignedMessage { payload: [66, 204, 6, 10, 34, 10, 32, 24, 184, 155, 6, 101, 21, 108, 170, 62, 153, 21, 118, 143, 175, 230, 48, 108, 236, 240, 51, 227, 40, 231, 10, 164, 230, 221, 71, 246, 49, 62, 144, 18, 190, 1, 16, 141, 236, 11, 34, 34, 10, 32, 108, 39, 182, 111, 12, 235, 101, 0, 148, 60, 81, 160, 251, 156, 64, 151, 53, 175, 229, 90, 32, 14, 103, 19, 233, 85, 53, 103, 251, 253, 153, 196, 42, 34, 10, 32, 198, 192, 170, 7, 242, 116, 147, 210, 242, 229, 207, 245, 108, 137, 10, 53, 58, 32, 8, 109, 108, 37, 236, 130, 81, 40, 225, 42, 231, 82, 178, 217, 50, 34, 10, 32, 42, 186, 142, 228, 197, 210, 94, 243, 154, 249, 108, 56, 220, 21, 32, 83, 154, 190, 249, 148, 104, 172, 71, 197, 100, 58, 170, 0, 224, 152, 4, 8, 58, 34, 10, 32, 115, 36, 181, 199, 43, 81, 187, 93, 76, 24, 15, 17, 9, 207, 211, 71, 182, 4, 115, 136, 33, 69, 132, 28, 57, 243, 229, 132, 87, 98, 150, 249, 66, 40, 10, 38, 10, 17, 10, 5, 101, 112, 111, 99, 104, 18, 8, 13, 246, 2, 0, 0, 0, 0, 0, 10, 17, 10, 11, 112, 114, 111, 112, 111, 115, 101, 114, 95, 105, 100, 18, 2, 2, 0, 26, 202, 1, 10, 96, 18, 94, 8, 1, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 241, 238, 158, 182, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 214, 203, 195, 9, 83, 214, 126, 250, 63, 194, 111, 153, 142, 215, 41, 135, 7, 183, 234, 78, 227, 113, 150, 104, 85, 17, 170, 81, 155, 202, 18, 75, 208, 175, 220, 206, 191, 142, 230, 218, 240, 152, 157, 145, 130, 15, 26, 122, 5, 6, 148, 200, 61, 170, 76, 49, 150, 138, 199, 189, 212, 86, 50, 4, 26, 202, 1, 10, 96, 18, 94, 8, 2, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 141, 228, 246, 176, 2, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 204, 115, 84, 55, 124, 16, 82, 4, 99, 139, 20, 13, 169, 244, 149, 209, 36, 62, 12, 5, 44, 1, 114, 235, 96, 182, 149, 225, 32, 22, 140, 158, 230, 140, 190, 203, 90, 95, 98, 216, 54, 60, 141, 178, 59, 67, 114, 40, 251, 234, 221, 136, 206, 249, 112, 72, 136, 220, 241, 217, 197, 147, 255, 4, 26, 202, 1, 10, 96, 18, 94, 8, 3, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 217, 155, 167, 233, 2, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 105, 240, 121, 221, 0, 44, 111, 169, 143, 156, 198, 41, 127, 171, 129, 117, 145, 76, 60, 160, 73, 189, 93, 231, 80, 70, 254, 237, 169, 53, 82, 114, 87, 55, 166, 70, 151, 80, 155, 83, 44, 15, 74, 206, 147, 52, 246, 192, 15, 6, 71, 224, 241, 107, 166, 237, 0, 6, 187, 58, 96, 96, 107, 14], author: PublicKey(1d7701dd...), signature: Signature(9e1922c8...) }, inner: BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194061), tx_count: 0, prev_hash: Hash(6c27b66f...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [13, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [2, 0]}) } }, precommits: [[10, 96, 18, 94, 8, 1, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 241, 238, 158, 182, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 214, 203, 195, 9, 83, 214, 126, 250, 63, 194, 111, 153, 142, 215, 41, 135, 7, 183, 234, 78, 227, 113, 150, 104, 85, 17, 170, 81, 155, 202, 18, 75, 208, 175, 220, 206, 191, 142, 230, 218, 240, 152, 157, 145, 130, 15, 26, 122, 5, 6, 148, 200, 61, 170, 76, 49, 150, 138, 199, 189, 212, 86, 50, 4], [10, 96, 18, 94, 8, 2, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 141, 228, 246, 176, 2, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 204, 115, 84, 55, 124, 16, 82, 4, 99, 139, 20, 13, 169, 244, 149, 209, 36, 62, 12, 5, 44, 1, 114, 235, 96, 182, 149, 225, 32, 22, 140, 158, 230, 140, 190, 203, 90, 95, 98, 216, 54, 60, 141, 178, 59, 67, 114, 40, 251, 234, 221, 136, 206, 249, 112, 72, 136, 220, 241, 217, 197, 147, 255, 4], [10, 96, 18, 94, 8, 3, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 217, 155, 167, 233, 2, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 105, 240, 121, 221, 0, 44, 111, 169, 143, 156, 198, 41, 127, 171, 129, 117, 145, 76, 60, 160, 73, 189, 93, 231, 80, 70, 254, 237, 169, 53, 82, 114, 87, 55, 166, 70, 151, 80, 155, 83, 44, 15, 74, 206, 147, 52, 246, 192, 15, 6, 71, 224, 241, 107, 166, 237, 0, 6, 187, 58, 96, 96, 107, 14]], transactions: [] } }
[2020-08-01T06:04:08.046038592Z ERROR exonum_node::consensus] Received incorrect block BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194061), tx_count: 0, prev_hash: Hash(6c27b66f...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [13, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [2, 0]}) } }, precommits: [[10, 96, 18, 94, 8, 3, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 217, 155, 167, 233, 2, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 105, 240, 121, 221, 0, 44, 111, 169, 143, 156, 198, 41, 127, 171, 129, 117, 145, 76, 60, 160, 73, 189, 93, 231, 80, 70, 254, 237, 169, 53, 82, 114, 87, 55, 166, 70, 151, 80, 155, 83, 44, 15, 74, 206, 147, 52, 246, 192, 15, 6, 71, 224, 241, 107, 166, 237, 0, 6, 187, 58, 96, 96, 107, 14], [10, 96, 18, 94, 8, 1, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 241, 238, 158, 182, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 214, 203, 195, 9, 83, 214, 126, 250, 63, 194, 111, 153, 142, 215, 41, 135, 7, 183, 234, 78, 227, 113, 150, 104, 85, 17, 170, 81, 155, 202, 18, 75, 208, 175, 220, 206, 191, 142, 230, 218, 240, 152, 157, 145, 130, 15, 26, 122, 5, 6, 148, 200, 61, 170, 76, 49, 150, 138, 199, 189, 212, 86, 50, 4], [10, 96, 18, 94, 8, 2, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 141, 228, 246, 176, 2, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 204, 115, 84, 55, 124, 16, 82, 4, 99, 139, 20, 13, 169, 244, 149, 209, 36, 62, 12, 5, 44, 1, 114, 235, 96, 182, 149, 225, 32, 22, 140, 158, 230, 140, 190, 203, 90, 95, 98, 216, 54, 60, 141, 178, 59, 67, 114, 40, 251, 234, 221, 136, 206, 249, 112, 72, 136, 220, 241, 217, 197, 147, 255, 4]], transactions: [] }: Received block has inappropriate height or epoch, msg=Verified { raw: SignedMessage { payload: [66, 204, 6, 10, 34, 10, 32, 24, 184, 155, 6, 101, 21, 108, 170, 62, 153, 21, 118, 143, 175, 230, 48, 108, 236, 240, 51, 227, 40, 231, 10, 164, 230, 221, 71, 246, 49, 62, 144, 18, 190, 1, 16, 141, 236, 11, 34, 34, 10, 32, 108, 39, 182, 111, 12, 235, 101, 0, 148, 60, 81, 160, 251, 156, 64, 151, 53, 175, 229, 90, 32, 14, 103, 19, 233, 85, 53, 103, 251, 253, 153, 196, 42, 34, 10, 32, 198, 192, 170, 7, 242, 116, 147, 210, 242, 229, 207, 245, 108, 137, 10, 53, 58, 32, 8, 109, 108, 37, 236, 130, 81, 40, 225, 42, 231, 82, 178, 217, 50, 34, 10, 32, 42, 186, 142, 228, 197, 210, 94, 243, 154, 249, 108, 56, 220, 21, 32, 83, 154, 190, 249, 148, 104, 172, 71, 197, 100, 58, 170, 0, 224, 152, 4, 8, 58, 34, 10, 32, 115, 36, 181, 199, 43, 81, 187, 93, 76, 24, 15, 17, 9, 207, 211, 71, 182, 4, 115, 136, 33, 69, 132, 28, 57, 243, 229, 132, 87, 98, 150, 249, 66, 40, 10, 38, 10, 17, 10, 5, 101, 112, 111, 99, 104, 18, 8, 13, 246, 2, 0, 0, 0, 0, 0, 10, 17, 10, 11, 112, 114, 111, 112, 111, 115, 101, 114, 95, 105, 100, 18, 2, 2, 0, 26, 202, 1, 10, 96, 18, 94, 8, 3, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 217, 155, 167, 233, 2, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 105, 240, 121, 221, 0, 44, 111, 169, 143, 156, 198, 41, 127, 171, 129, 117, 145, 76, 60, 160, 73, 189, 93, 231, 80, 70, 254, 237, 169, 53, 82, 114, 87, 55, 166, 70, 151, 80, 155, 83, 44, 15, 74, 206, 147, 52, 246, 192, 15, 6, 71, 224, 241, 107, 166, 237, 0, 6, 187, 58, 96, 96, 107, 14, 26, 202, 1, 10, 96, 18, 94, 8, 1, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 241, 238, 158, 182, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 214, 203, 195, 9, 83, 214, 126, 250, 63, 194, 111, 153, 142, 215, 41, 135, 7, 183, 234, 78, 227, 113, 150, 104, 85, 17, 170, 81, 155, 202, 18, 75, 208, 175, 220, 206, 191, 142, 230, 218, 240, 152, 157, 145, 130, 15, 26, 122, 5, 6, 148, 200, 61, 170, 76, 49, 150, 138, 199, 189, 212, 86, 50, 4, 26, 202, 1, 10, 96, 18, 94, 8, 2, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 141, 228, 246, 176, 2, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 204, 115, 84, 55, 124, 16, 82, 4, 99, 139, 20, 13, 169, 244, 149, 209, 36, 62, 12, 5, 44, 1, 114, 235, 96, 182, 149, 225, 32, 22, 140, 158, 230, 140, 190, 203, 90, 95, 98, 216, 54, 60, 141, 178, 59, 67, 114, 40, 251, 234, 221, 136, 206, 249, 112, 72, 136, 220, 241, 217, 197, 147, 255, 4], author: PublicKey(be00c0dc...), signature: Signature(8743ced7...) }, inner: BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194061), tx_count: 0, prev_hash: Hash(6c27b66f...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [13, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [2, 0]}) } }, precommits: [[10, 96, 18, 94, 8, 3, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 217, 155, 167, 233, 2, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 105, 240, 121, 221, 0, 44, 111, 169, 143, 156, 198, 41, 127, 171, 129, 117, 145, 76, 60, 160, 73, 189, 93, 231, 80, 70, 254, 237, 169, 53, 82, 114, 87, 55, 166, 70, 151, 80, 155, 83, 44, 15, 74, 206, 147, 52, 246, 192, 15, 6, 71, 224, 241, 107, 166, 237, 0, 6, 187, 58, 96, 96, 107, 14], [10, 96, 18, 94, 8, 1, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 241, 238, 158, 182, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 214, 203, 195, 9, 83, 214, 126, 250, 63, 194, 111, 153, 142, 215, 41, 135, 7, 183, 234, 78, 227, 113, 150, 104, 85, 17, 170, 81, 155, 202, 18, 75, 208, 175, 220, 206, 191, 142, 230, 218, 240, 152, 157, 145, 130, 15, 26, 122, 5, 6, 148, 200, 61, 170, 76, 49, 150, 138, 199, 189, 212, 86, 50, 4], [10, 96, 18, 94, 8, 2, 16, 141, 236, 11, 24, 1, 34, 34, 10, 32, 87, 171, 182, 142, 6, 57, 90, 196, 234, 10, 109, 92, 207, 194, 53, 238, 77, 150, 220, 59, 58, 34, 161, 133, 246, 134, 188, 22, 32, 224, 157, 98, 42, 34, 10, 32, 225, 253, 255, 78, 52, 0, 47, 202, 207, 13, 54, 167, 224, 46, 199, 210, 101, 226, 167, 153, 117, 82, 51, 8, 119, 16, 94, 82, 115, 21, 194, 132, 50, 12, 8, 166, 139, 148, 249, 5, 16, 141, 228, 246, 176, 2, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 204, 115, 84, 55, 124, 16, 82, 4, 99, 139, 20, 13, 169, 244, 149, 209, 36, 62, 12, 5, 44, 1, 114, 235, 96, 182, 149, 225, 32, 22, 140, 158, 230, 140, 190, 203, 90, 95, 98, 216, 54, 60, 141, 178, 59, 67, 114, 40, 251, 234, 221, 136, 206, 249, 112, 72, 136, 220, 241, 217, 197, 147, 255, 4]], transactions: [] } }
[2020-08-01T06:04:35.966595935Z ERROR exonum_node::consensus] Received incorrect block BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194063), tx_count: 0, prev_hash: Hash(7ca8a1e2...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [15, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [1, 0]}) } }, precommits: [[10, 96, 18, 94, 8, 1, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 142, 136, 153, 156, 3, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 248, 46, 15, 213, 95, 139, 45, 71, 242, 158, 15, 71, 26, 79, 65, 71, 79, 250, 72, 94, 138, 252, 186, 156, 168, 238, 51, 38, 65, 155, 100, 18, 24, 155, 40, 99, 71, 30, 87, 159, 26, 128, 230, 45, 249, 242, 204, 226, 64, 11, 213, 185, 128, 186, 74, 32, 73, 237, 243, 65, 54, 116, 144, 11], [10, 96, 18, 94, 8, 2, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 249, 158, 162, 128, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 133, 118, 174, 132, 1, 204, 18, 149, 100, 19, 13, 107, 73, 182, 204, 89, 51, 118, 46, 132, 58, 230, 253, 249, 150, 48, 80, 176, 222, 81, 190, 18, 253, 189, 213, 83, 166, 214, 240, 60, 198, 10, 182, 12, 40, 174, 64, 45, 113, 198, 254, 185, 159, 244, 180, 51, 91, 171, 82, 136, 107, 215, 229, 2], [10, 96, 18, 94, 8, 3, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 248, 175, 181, 219, 3, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 122, 118, 123, 244, 244, 187, 95, 183, 201, 52, 214, 177, 147, 218, 156, 219, 184, 141, 73, 225, 81, 110, 175, 99, 85, 230, 226, 119, 152, 110, 244, 203, 10, 115, 44, 73, 161, 190, 7, 30, 200, 17, 150, 174, 146, 91, 44, 151, 177, 206, 60, 55, 177, 227, 32, 98, 216, 106, 152, 67, 160, 18, 39, 2]], transactions: [] }: Received block has inappropriate height or epoch, msg=Verified { raw: SignedMessage { payload: [66, 204, 6, 10, 34, 10, 32, 24, 184, 155, 6, 101, 21, 108, 170, 62, 153, 21, 118, 143, 175, 230, 48, 108, 236, 240, 51, 227, 40, 231, 10, 164, 230, 221, 71, 246, 49, 62, 144, 18, 190, 1, 16, 143, 236, 11, 34, 34, 10, 32, 124, 168, 161, 226, 84, 210, 113, 49, 210, 230, 191, 63, 7, 149, 126, 0, 105, 193, 163, 179, 151, 120, 188, 162, 189, 220, 211, 86, 226, 139, 45, 140, 42, 34, 10, 32, 198, 192, 170, 7, 242, 116, 147, 210, 242, 229, 207, 245, 108, 137, 10, 53, 58, 32, 8, 109, 108, 37, 236, 130, 81, 40, 225, 42, 231, 82, 178, 217, 50, 34, 10, 32, 42, 186, 142, 228, 197, 210, 94, 243, 154, 249, 108, 56, 220, 21, 32, 83, 154, 190, 249, 148, 104, 172, 71, 197, 100, 58, 170, 0, 224, 152, 4, 8, 58, 34, 10, 32, 115, 36, 181, 199, 43, 81, 187, 93, 76, 24, 15, 17, 9, 207, 211, 71, 182, 4, 115, 136, 33, 69, 132, 28, 57, 243, 229, 132, 87, 98, 150, 249, 66, 40, 10, 38, 10, 17, 10, 5, 101, 112, 111, 99, 104, 18, 8, 15, 246, 2, 0, 0, 0, 0, 0, 10, 17, 10, 11, 112, 114, 111, 112, 111, 115, 101, 114, 95, 105, 100, 18, 2, 1, 0, 26, 202, 1, 10, 96, 18, 94, 8, 1, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 142, 136, 153, 156, 3, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 248, 46, 15, 213, 95, 139, 45, 71, 242, 158, 15, 71, 26, 79, 65, 71, 79, 250, 72, 94, 138, 252, 186, 156, 168, 238, 51, 38, 65, 155, 100, 18, 24, 155, 40, 99, 71, 30, 87, 159, 26, 128, 230, 45, 249, 242, 204, 226, 64, 11, 213, 185, 128, 186, 74, 32, 73, 237, 243, 65, 54, 116, 144, 11, 26, 202, 1, 10, 96, 18, 94, 8, 2, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 249, 158, 162, 128, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 133, 118, 174, 132, 1, 204, 18, 149, 100, 19, 13, 107, 73, 182, 204, 89, 51, 118, 46, 132, 58, 230, 253, 249, 150, 48, 80, 176, 222, 81, 190, 18, 253, 189, 213, 83, 166, 214, 240, 60, 198, 10, 182, 12, 40, 174, 64, 45, 113, 198, 254, 185, 159, 244, 180, 51, 91, 171, 82, 136, 107, 215, 229, 2, 26, 202, 1, 10, 96, 18, 94, 8, 3, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 248, 175, 181, 219, 3, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 122, 118, 123, 244, 244, 187, 95, 183, 201, 52, 214, 177, 147, 218, 156, 219, 184, 141, 73, 225, 81, 110, 175, 99, 85, 230, 226, 119, 152, 110, 244, 203, 10, 115, 44, 73, 161, 190, 7, 30, 200, 17, 150, 174, 146, 91, 44, 151, 177, 206, 60, 55, 177, 227, 32, 98, 216, 106, 152, 67, 160, 18, 39, 2], author: PublicKey(1d7701dd...), signature: Signature(5cde7edc...) }, inner: BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194063), tx_count: 0, prev_hash: Hash(7ca8a1e2...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [15, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [1, 0]}) } }, precommits: [[10, 96, 18, 94, 8, 1, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 142, 136, 153, 156, 3, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 248, 46, 15, 213, 95, 139, 45, 71, 242, 158, 15, 71, 26, 79, 65, 71, 79, 250, 72, 94, 138, 252, 186, 156, 168, 238, 51, 38, 65, 155, 100, 18, 24, 155, 40, 99, 71, 30, 87, 159, 26, 128, 230, 45, 249, 242, 204, 226, 64, 11, 213, 185, 128, 186, 74, 32, 73, 237, 243, 65, 54, 116, 144, 11], [10, 96, 18, 94, 8, 2, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 249, 158, 162, 128, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 133, 118, 174, 132, 1, 204, 18, 149, 100, 19, 13, 107, 73, 182, 204, 89, 51, 118, 46, 132, 58, 230, 253, 249, 150, 48, 80, 176, 222, 81, 190, 18, 253, 189, 213, 83, 166, 214, 240, 60, 198, 10, 182, 12, 40, 174, 64, 45, 113, 198, 254, 185, 159, 244, 180, 51, 91, 171, 82, 136, 107, 215, 229, 2], [10, 96, 18, 94, 8, 3, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 248, 175, 181, 219, 3, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 122, 118, 123, 244, 244, 187, 95, 183, 201, 52, 214, 177, 147, 218, 156, 219, 184, 141, 73, 225, 81, 110, 175, 99, 85, 230, 226, 119, 152, 110, 244, 203, 10, 115, 44, 73, 161, 190, 7, 30, 200, 17, 150, 174, 146, 91, 44, 151, 177, 206, 60, 55, 177, 227, 32, 98, 216, 106, 152, 67, 160, 18, 39, 2]], transactions: [] } }
[2020-08-01T06:04:35.991448190Z ERROR exonum_node::consensus] Received incorrect block BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194063), tx_count: 0, prev_hash: Hash(7ca8a1e2...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [15, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [1, 0]}) } }, precommits: [[10, 96, 18, 94, 8, 3, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 248, 175, 181, 219, 3, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 122, 118, 123, 244, 244, 187, 95, 183, 201, 52, 214, 177, 147, 218, 156, 219, 184, 141, 73, 225, 81, 110, 175, 99, 85, 230, 226, 119, 152, 110, 244, 203, 10, 115, 44, 73, 161, 190, 7, 30, 200, 17, 150, 174, 146, 91, 44, 151, 177, 206, 60, 55, 177, 227, 32, 98, 216, 106, 152, 67, 160, 18, 39, 2], [10, 96, 18, 94, 8, 2, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 249, 158, 162, 128, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 133, 118, 174, 132, 1, 204, 18, 149, 100, 19, 13, 107, 73, 182, 204, 89, 51, 118, 46, 132, 58, 230, 253, 249, 150, 48, 80, 176, 222, 81, 190, 18, 253, 189, 213, 83, 166, 214, 240, 60, 198, 10, 182, 12, 40, 174, 64, 45, 113, 198, 254, 185, 159, 244, 180, 51, 91, 171, 82, 136, 107, 215, 229, 2], [10, 96, 18, 94, 8, 1, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 142, 136, 153, 156, 3, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 248, 46, 15, 213, 95, 139, 45, 71, 242, 158, 15, 71, 26, 79, 65, 71, 79, 250, 72, 94, 138, 252, 186, 156, 168, 238, 51, 38, 65, 155, 100, 18, 24, 155, 40, 99, 71, 30, 87, 159, 26, 128, 230, 45, 249, 242, 204, 226, 64, 11, 213, 185, 128, 186, 74, 32, 73, 237, 243, 65, 54, 116, 144, 11]], transactions: [] }: Received block has inappropriate height or epoch, msg=Verified { raw: SignedMessage { payload: [66, 204, 6, 10, 34, 10, 32, 24, 184, 155, 6, 101, 21, 108, 170, 62, 153, 21, 118, 143, 175, 230, 48, 108, 236, 240, 51, 227, 40, 231, 10, 164, 230, 221, 71, 246, 49, 62, 144, 18, 190, 1, 16, 143, 236, 11, 34, 34, 10, 32, 124, 168, 161, 226, 84, 210, 113, 49, 210, 230, 191, 63, 7, 149, 126, 0, 105, 193, 163, 179, 151, 120, 188, 162, 189, 220, 211, 86, 226, 139, 45, 140, 42, 34, 10, 32, 198, 192, 170, 7, 242, 116, 147, 210, 242, 229, 207, 245, 108, 137, 10, 53, 58, 32, 8, 109, 108, 37, 236, 130, 81, 40, 225, 42, 231, 82, 178, 217, 50, 34, 10, 32, 42, 186, 142, 228, 197, 210, 94, 243, 154, 249, 108, 56, 220, 21, 32, 83, 154, 190, 249, 148, 104, 172, 71, 197, 100, 58, 170, 0, 224, 152, 4, 8, 58, 34, 10, 32, 115, 36, 181, 199, 43, 81, 187, 93, 76, 24, 15, 17, 9, 207, 211, 71, 182, 4, 115, 136, 33, 69, 132, 28, 57, 243, 229, 132, 87, 98, 150, 249, 66, 40, 10, 38, 10, 17, 10, 5, 101, 112, 111, 99, 104, 18, 8, 15, 246, 2, 0, 0, 0, 0, 0, 10, 17, 10, 11, 112, 114, 111, 112, 111, 115, 101, 114, 95, 105, 100, 18, 2, 1, 0, 26, 202, 1, 10, 96, 18, 94, 8, 3, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 248, 175, 181, 219, 3, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 122, 118, 123, 244, 244, 187, 95, 183, 201, 52, 214, 177, 147, 218, 156, 219, 184, 141, 73, 225, 81, 110, 175, 99, 85, 230, 226, 119, 152, 110, 244, 203, 10, 115, 44, 73, 161, 190, 7, 30, 200, 17, 150, 174, 146, 91, 44, 151, 177, 206, 60, 55, 177, 227, 32, 98, 216, 106, 152, 67, 160, 18, 39, 2, 26, 202, 1, 10, 96, 18, 94, 8, 2, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 249, 158, 162, 128, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 133, 118, 174, 132, 1, 204, 18, 149, 100, 19, 13, 107, 73, 182, 204, 89, 51, 118, 46, 132, 58, 230, 253, 249, 150, 48, 80, 176, 222, 81, 190, 18, 253, 189, 213, 83, 166, 214, 240, 60, 198, 10, 182, 12, 40, 174, 64, 45, 113, 198, 254, 185, 159, 244, 180, 51, 91, 171, 82, 136, 107, 215, 229, 2, 26, 202, 1, 10, 96, 18, 94, 8, 1, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 142, 136, 153, 156, 3, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 248, 46, 15, 213, 95, 139, 45, 71, 242, 158, 15, 71, 26, 79, 65, 71, 79, 250, 72, 94, 138, 252, 186, 156, 168, 238, 51, 38, 65, 155, 100, 18, 24, 155, 40, 99, 71, 30, 87, 159, 26, 128, 230, 45, 249, 242, 204, 226, 64, 11, 213, 185, 128, 186, 74, 32, 73, 237, 243, 65, 54, 116, 144, 11], author: PublicKey(be00c0dc...), signature: Signature(f21ef8b0...) }, inner: BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194063), tx_count: 0, prev_hash: Hash(7ca8a1e2...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [15, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [1, 0]}) } }, precommits: [[10, 96, 18, 94, 8, 3, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 248, 175, 181, 219, 3, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 122, 118, 123, 244, 244, 187, 95, 183, 201, 52, 214, 177, 147, 218, 156, 219, 184, 141, 73, 225, 81, 110, 175, 99, 85, 230, 226, 119, 152, 110, 244, 203, 10, 115, 44, 73, 161, 190, 7, 30, 200, 17, 150, 174, 146, 91, 44, 151, 177, 206, 60, 55, 177, 227, 32, 98, 216, 106, 152, 67, 160, 18, 39, 2], [10, 96, 18, 94, 8, 2, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 249, 158, 162, 128, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 133, 118, 174, 132, 1, 204, 18, 149, 100, 19, 13, 107, 73, 182, 204, 89, 51, 118, 46, 132, 58, 230, 253, 249, 150, 48, 80, 176, 222, 81, 190, 18, 253, 189, 213, 83, 166, 214, 240, 60, 198, 10, 182, 12, 40, 174, 64, 45, 113, 198, 254, 185, 159, 244, 180, 51, 91, 171, 82, 136, 107, 215, 229, 2], [10, 96, 18, 94, 8, 1, 16, 143, 236, 11, 24, 2, 34, 34, 10, 32, 7, 32, 120, 162, 23, 123, 246, 184, 104, 30, 88, 4, 67, 80, 94, 43, 179, 222, 162, 155, 196, 142, 23, 32, 202, 252, 235, 163, 174, 56, 46, 9, 42, 34, 10, 32, 127, 61, 232, 153, 183, 211, 38, 160, 86, 250, 188, 211, 73, 149, 123, 5, 109, 123, 59, 102, 117, 123, 197, 119, 204, 36, 109, 100, 157, 212, 7, 212, 50, 12, 8, 192, 139, 148, 249, 5, 16, 142, 136, 153, 156, 3, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 248, 46, 15, 213, 95, 139, 45, 71, 242, 158, 15, 71, 26, 79, 65, 71, 79, 250, 72, 94, 138, 252, 186, 156, 168, 238, 51, 38, 65, 155, 100, 18, 24, 155, 40, 99, 71, 30, 87, 159, 26, 128, 230, 45, 249, 242, 204, 226, 64, 11, 213, 185, 128, 186, 74, 32, 73, 237, 243, 65, 54, 116, 144, 11]], transactions: [] } }
[2020-08-01T06:05:02.270108509Z ERROR exonum_node::consensus] Received incorrect block BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194065), tx_count: 0, prev_hash: Hash(9cddb8b2...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [17, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [2, 0]}) } }, precommits: [[10, 95, 18, 93, 8, 1, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 161, 129, 128, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 121, 75, 140, 141, 95, 169, 18, 229, 14, 63, 20, 213, 27, 69, 69, 222, 122, 215, 125, 186, 173, 134, 23, 23, 64, 5, 28, 218, 27, 214, 129, 124, 146, 25, 106, 164, 128, 156, 106, 205, 20, 160, 249, 159, 203, 30, 156, 70, 93, 71, 168, 93, 255, 67, 73, 85, 64, 232, 161, 18, 115, 127, 241, 12], [10, 96, 18, 94, 8, 2, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 12, 8, 215, 139, 148, 249, 5, 16, 203, 154, 180, 205, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 119, 103, 103, 42, 218, 255, 240, 51, 141, 113, 172, 135, 212, 82, 147, 243, 78, 245, 233, 96, 121, 84, 74, 225, 95, 131, 162, 64, 34, 89, 12, 188, 222, 202, 130, 235, 42, 204, 31, 197, 71, 3, 32, 66, 20, 11, 212, 101, 85, 28, 137, 250, 224, 160, 68, 62, 172, 115, 166, 73, 87, 5, 186, 13], [10, 95, 18, 93, 8, 3, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 163, 253, 145, 60, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 188, 241, 67, 215, 73, 151, 161, 171, 93, 33, 202, 120, 118, 71, 196, 242, 39, 17, 39, 113, 62, 161, 49, 59, 56, 191, 167, 211, 24, 154, 132, 203, 124, 164, 35, 125, 207, 91, 27, 192, 94, 166, 96, 90, 139, 159, 199, 74, 159, 187, 15, 157, 142, 233, 202, 234, 191, 121, 62, 199, 140, 96, 108, 1]], transactions: [] }: Received block has inappropriate height or epoch, msg=Verified { raw: SignedMessage { payload: [66, 202, 6, 10, 34, 10, 32, 24, 184, 155, 6, 101, 21, 108, 170, 62, 153, 21, 118, 143, 175, 230, 48, 108, 236, 240, 51, 227, 40, 231, 10, 164, 230, 221, 71, 246, 49, 62, 144, 18, 190, 1, 16, 145, 236, 11, 34, 34, 10, 32, 156, 221, 184, 178, 216, 167, 168, 32, 154, 76, 160, 33, 5, 47, 115, 12, 160, 45, 50, 158, 9, 254, 100, 28, 163, 245, 64, 25, 121, 12, 218, 71, 42, 34, 10, 32, 198, 192, 170, 7, 242, 116, 147, 210, 242, 229, 207, 245, 108, 137, 10, 53, 58, 32, 8, 109, 108, 37, 236, 130, 81, 40, 225, 42, 231, 82, 178, 217, 50, 34, 10, 32, 42, 186, 142, 228, 197, 210, 94, 243, 154, 249, 108, 56, 220, 21, 32, 83, 154, 190, 249, 148, 104, 172, 71, 197, 100, 58, 170, 0, 224, 152, 4, 8, 58, 34, 10, 32, 115, 36, 181, 199, 43, 81, 187, 93, 76, 24, 15, 17, 9, 207, 211, 71, 182, 4, 115, 136, 33, 69, 132, 28, 57, 243, 229, 132, 87, 98, 150, 249, 66, 40, 10, 38, 10, 17, 10, 5, 101, 112, 111, 99, 104, 18, 8, 17, 246, 2, 0, 0, 0, 0, 0, 10, 17, 10, 11, 112, 114, 111, 112, 111, 115, 101, 114, 95, 105, 100, 18, 2, 2, 0, 26, 201, 1, 10, 95, 18, 93, 8, 1, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 161, 129, 128, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 121, 75, 140, 141, 95, 169, 18, 229, 14, 63, 20, 213, 27, 69, 69, 222, 122, 215, 125, 186, 173, 134, 23, 23, 64, 5, 28, 218, 27, 214, 129, 124, 146, 25, 106, 164, 128, 156, 106, 205, 20, 160, 249, 159, 203, 30, 156, 70, 93, 71, 168, 93, 255, 67, 73, 85, 64, 232, 161, 18, 115, 127, 241, 12, 26, 202, 1, 10, 96, 18, 94, 8, 2, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 12, 8, 215, 139, 148, 249, 5, 16, 203, 154, 180, 205, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 119, 103, 103, 42, 218, 255, 240, 51, 141, 113, 172, 135, 212, 82, 147, 243, 78, 245, 233, 96, 121, 84, 74, 225, 95, 131, 162, 64, 34, 89, 12, 188, 222, 202, 130, 235, 42, 204, 31, 197, 71, 3, 32, 66, 20, 11, 212, 101, 85, 28, 137, 250, 224, 160, 68, 62, 172, 115, 166, 73, 87, 5, 186, 13, 26, 201, 1, 10, 95, 18, 93, 8, 3, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 163, 253, 145, 60, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 188, 241, 67, 215, 73, 151, 161, 171, 93, 33, 202, 120, 118, 71, 196, 242, 39, 17, 39, 113, 62, 161, 49, 59, 56, 191, 167, 211, 24, 154, 132, 203, 124, 164, 35, 125, 207, 91, 27, 192, 94, 166, 96, 90, 139, 159, 199, 74, 159, 187, 15, 157, 142, 233, 202, 234, 191, 121, 62, 199, 140, 96, 108, 1], author: PublicKey(1d7701dd...), signature: Signature(6e79a33a...) }, inner: BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194065), tx_count: 0, prev_hash: Hash(9cddb8b2...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [17, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [2, 0]}) } }, precommits: [[10, 95, 18, 93, 8, 1, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 161, 129, 128, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 121, 75, 140, 141, 95, 169, 18, 229, 14, 63, 20, 213, 27, 69, 69, 222, 122, 215, 125, 186, 173, 134, 23, 23, 64, 5, 28, 218, 27, 214, 129, 124, 146, 25, 106, 164, 128, 156, 106, 205, 20, 160, 249, 159, 203, 30, 156, 70, 93, 71, 168, 93, 255, 67, 73, 85, 64, 232, 161, 18, 115, 127, 241, 12], [10, 96, 18, 94, 8, 2, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 12, 8, 215, 139, 148, 249, 5, 16, 203, 154, 180, 205, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 119, 103, 103, 42, 218, 255, 240, 51, 141, 113, 172, 135, 212, 82, 147, 243, 78, 245, 233, 96, 121, 84, 74, 225, 95, 131, 162, 64, 34, 89, 12, 188, 222, 202, 130, 235, 42, 204, 31, 197, 71, 3, 32, 66, 20, 11, 212, 101, 85, 28, 137, 250, 224, 160, 68, 62, 172, 115, 166, 73, 87, 5, 186, 13], [10, 95, 18, 93, 8, 3, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 163, 253, 145, 60, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 188, 241, 67, 215, 73, 151, 161, 171, 93, 33, 202, 120, 118, 71, 196, 242, 39, 17, 39, 113, 62, 161, 49, 59, 56, 191, 167, 211, 24, 154, 132, 203, 124, 164, 35, 125, 207, 91, 27, 192, 94, 166, 96, 90, 139, 159, 199, 74, 159, 187, 15, 157, 142, 233, 202, 234, 191, 121, 62, 199, 140, 96, 108, 1]], transactions: [] } }
[2020-08-01T06:05:02.315904935Z ERROR exonum_node::consensus] Received incorrect block BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194065), tx_count: 0, prev_hash: Hash(9cddb8b2...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [17, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [2, 0]}) } }, precommits: [[10, 95, 18, 93, 8, 3, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 163, 253, 145, 60, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 188, 241, 67, 215, 73, 151, 161, 171, 93, 33, 202, 120, 118, 71, 196, 242, 39, 17, 39, 113, 62, 161, 49, 59, 56, 191, 167, 211, 24, 154, 132, 203, 124, 164, 35, 125, 207, 91, 27, 192, 94, 166, 96, 90, 139, 159, 199, 74, 159, 187, 15, 157, 142, 233, 202, 234, 191, 121, 62, 199, 140, 96, 108, 1], [10, 96, 18, 94, 8, 2, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 12, 8, 215, 139, 148, 249, 5, 16, 203, 154, 180, 205, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 119, 103, 103, 42, 218, 255, 240, 51, 141, 113, 172, 135, 212, 82, 147, 243, 78, 245, 233, 96, 121, 84, 74, 225, 95, 131, 162, 64, 34, 89, 12, 188, 222, 202, 130, 235, 42, 204, 31, 197, 71, 3, 32, 66, 20, 11, 212, 101, 85, 28, 137, 250, 224, 160, 68, 62, 172, 115, 166, 73, 87, 5, 186, 13], [10, 95, 18, 93, 8, 1, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 161, 129, 128, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 121, 75, 140, 141, 95, 169, 18, 229, 14, 63, 20, 213, 27, 69, 69, 222, 122, 215, 125, 186, 173, 134, 23, 23, 64, 5, 28, 218, 27, 214, 129, 124, 146, 25, 106, 164, 128, 156, 106, 205, 20, 160, 249, 159, 203, 30, 156, 70, 93, 71, 168, 93, 255, 67, 73, 85, 64, 232, 161, 18, 115, 127, 241, 12]], transactions: [] }: Received block has inappropriate height or epoch, msg=Verified { raw: SignedMessage { payload: [66, 202, 6, 10, 34, 10, 32, 24, 184, 155, 6, 101, 21, 108, 170, 62, 153, 21, 118, 143, 175, 230, 48, 108, 236, 240, 51, 227, 40, 231, 10, 164, 230, 221, 71, 246, 49, 62, 144, 18, 190, 1, 16, 145, 236, 11, 34, 34, 10, 32, 156, 221, 184, 178, 216, 167, 168, 32, 154, 76, 160, 33, 5, 47, 115, 12, 160, 45, 50, 158, 9, 254, 100, 28, 163, 245, 64, 25, 121, 12, 218, 71, 42, 34, 10, 32, 198, 192, 170, 7, 242, 116, 147, 210, 242, 229, 207, 245, 108, 137, 10, 53, 58, 32, 8, 109, 108, 37, 236, 130, 81, 40, 225, 42, 231, 82, 178, 217, 50, 34, 10, 32, 42, 186, 142, 228, 197, 210, 94, 243, 154, 249, 108, 56, 220, 21, 32, 83, 154, 190, 249, 148, 104, 172, 71, 197, 100, 58, 170, 0, 224, 152, 4, 8, 58, 34, 10, 32, 115, 36, 181, 199, 43, 81, 187, 93, 76, 24, 15, 17, 9, 207, 211, 71, 182, 4, 115, 136, 33, 69, 132, 28, 57, 243, 229, 132, 87, 98, 150, 249, 66, 40, 10, 38, 10, 17, 10, 5, 101, 112, 111, 99, 104, 18, 8, 17, 246, 2, 0, 0, 0, 0, 0, 10, 17, 10, 11, 112, 114, 111, 112, 111, 115, 101, 114, 95, 105, 100, 18, 2, 2, 0, 26, 201, 1, 10, 95, 18, 93, 8, 3, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 163, 253, 145, 60, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 188, 241, 67, 215, 73, 151, 161, 171, 93, 33, 202, 120, 118, 71, 196, 242, 39, 17, 39, 113, 62, 161, 49, 59, 56, 191, 167, 211, 24, 154, 132, 203, 124, 164, 35, 125, 207, 91, 27, 192, 94, 166, 96, 90, 139, 159, 199, 74, 159, 187, 15, 157, 142, 233, 202, 234, 191, 121, 62, 199, 140, 96, 108, 1, 26, 202, 1, 10, 96, 18, 94, 8, 2, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 12, 8, 215, 139, 148, 249, 5, 16, 203, 154, 180, 205, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 119, 103, 103, 42, 218, 255, 240, 51, 141, 113, 172, 135, 212, 82, 147, 243, 78, 245, 233, 96, 121, 84, 74, 225, 95, 131, 162, 64, 34, 89, 12, 188, 222, 202, 130, 235, 42, 204, 31, 197, 71, 3, 32, 66, 20, 11, 212, 101, 85, 28, 137, 250, 224, 160, 68, 62, 172, 115, 166, 73, 87, 5, 186, 13, 26, 201, 1, 10, 95, 18, 93, 8, 1, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 161, 129, 128, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 121, 75, 140, 141, 95, 169, 18, 229, 14, 63, 20, 213, 27, 69, 69, 222, 122, 215, 125, 186, 173, 134, 23, 23, 64, 5, 28, 218, 27, 214, 129, 124, 146, 25, 106, 164, 128, 156, 106, 205, 20, 160, 249, 159, 203, 30, 156, 70, 93, 71, 168, 93, 255, 67, 73, 85, 64, 232, 161, 18, 115, 127, 241, 12], author: PublicKey(be00c0dc...), signature: Signature(09c0bc0b...) }, inner: BlockResponse { to: PublicKey(18b89b06...), block: Block { height: Height(194065), tx_count: 0, prev_hash: Hash(9cddb8b2...), tx_hash: Hash(c6c0aa07...), state_hash: Hash(2aba8ee4...), error_hash: Hash(7324b5c7...), additional_headers: AdditionalHeaders { headers: OrderedMap({"epoch": [17, 246, 2, 0, 0, 0, 0, 0], "proposer_id": [2, 0]}) } }, precommits: [[10, 95, 18, 93, 8, 3, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 163, 253, 145, 60, 18, 34, 10, 32, 190, 0, 192, 220, 246, 87, 63, 52, 139, 191, 186, 207, 220, 96, 157, 89, 112, 218, 139, 142, 91, 11, 89, 95, 156, 172, 160, 35, 255, 44, 156, 125, 26, 66, 10, 64, 188, 241, 67, 215, 73, 151, 161, 171, 93, 33, 202, 120, 118, 71, 196, 242, 39, 17, 39, 113, 62, 161, 49, 59, 56, 191, 167, 211, 24, 154, 132, 203, 124, 164, 35, 125, 207, 91, 27, 192, 94, 166, 96, 90, 139, 159, 199, 74, 159, 187, 15, 157, 142, 233, 202, 234, 191, 121, 62, 199, 140, 96, 108, 1], [10, 96, 18, 94, 8, 2, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 12, 8, 215, 139, 148, 249, 5, 16, 203, 154, 180, 205, 3, 18, 34, 10, 32, 163, 105, 123, 148, 142, 250, 1, 37, 9, 43, 187, 166, 163, 84, 205, 63, 33, 93, 105, 49, 24, 219, 175, 162, 139, 82, 23, 40, 20, 1, 211, 22, 26, 66, 10, 64, 119, 103, 103, 42, 218, 255, 240, 51, 141, 113, 172, 135, 212, 82, 147, 243, 78, 245, 233, 96, 121, 84, 74, 225, 95, 131, 162, 64, 34, 89, 12, 188, 222, 202, 130, 235, 42, 204, 31, 197, 71, 3, 32, 66, 20, 11, 212, 101, 85, 28, 137, 250, 224, 160, 68, 62, 172, 115, 166, 73, 87, 5, 186, 13], [10, 95, 18, 93, 8, 1, 16, 145, 236, 11, 24, 1, 34, 34, 10, 32, 248, 231, 50, 213, 81, 94, 1, 8, 77, 166, 33, 15, 25, 203, 4, 185, 144, 72, 13, 44, 103, 73, 40, 71, 29, 42, 9, 203, 158, 178, 201, 22, 42, 34, 10, 32, 231, 25, 179, 221, 255, 91, 4, 139, 34, 156, 56, 134, 72, 228, 126, 193, 85, 213, 209, 134, 138, 75, 91, 130, 44, 167, 146, 38, 115, 110, 69, 170, 50, 11, 8, 216, 139, 148, 249, 5, 16, 161, 129, 128, 2, 18, 34, 10, 32, 29, 119, 1, 221, 127, 150, 159, 208, 228, 185, 3, 187, 113, 29, 96, 181, 78, 188, 33, 113, 101, 216, 169, 176, 220, 114, 206, 182, 210, 110, 214, 224, 26, 66, 10, 64, 121, 75, 140, 141, 95, 169, 18, 229, 14, 63, 20, 213, 27, 69, 69, 222, 122, 215, 125, 186, 173, 134, 23, 23, 64, 5, 28, 218, 27, 214, 129, 124, 146, 25, 106, 164, 128, 156, 106, 205, 20, 160, 249, 159, 203, 30, 156, 70, 93, 71, 168, 93, 255, 67, 73, 85, 64, 232, 161, 18, 115, 127, 241, 12]], transactions: [] } }
@aleksuss Do you have any idea how to debug this issue? It is constantly reproduced and blocks us from going production. Any help will be highly appreciated.
Have you faced such an issue before on long-living networks?
No. I haven't. What I can recommend is to run a network without deploying any service. If the issue wouldn't repeat then there is a problem in a logic of your service.
Ok, thanks!
Hi @aleksuss ,
I've looked into this a bit closer.
Here is the local network of two exonum_cryptocurrency_advanced nodes. The first graph represents the height of the network ower the time, the second one shows blocks per minute (bpm) ower the time.
max_propose_timeout is 200 so the target value for commits per minute is 300 (1000ms * 60 / 200ms). We can see that right after starting the real value is close to 300 but then it slows down logarithmically.
A restart of nodes (~14:35 on the chart) fixes the issue but then network starts slowing down again.
There are no transactions in the ledger. We tested exonum_cryptocurrency_advanced network in AWS and results are the same. What can be the reason for such behaviour?
What I can recommend is to run a network without deploying any service. If the issue wouldn't repeat then there is a problem in a logic of your service.
Seems the issue is quite easy to reproduce with out of the box exonum_cryptocurrency_advanced example and it is related to Exonum core, not to any custom smart contract/ledger logic.
@aleksuss Do you have any plans to look to this issue? Do you need any additional info? It seems blocker to any production deployments.
Hello. Thanks for the input. I will think about such behavior but for now I think it's expected behavior. The reason of such behavior are proof indexes (collections). As you know, proof indexes use binary tree under the hood and use crypto functions for key calculations and other verifications. Even in you case (without transactions) indexes grow up (e.g. an index with blocks and other service indexes). So the bigger size of proof index, the bigger complexity to make some verifications. There is why amount of committed blocks decreses with time.
Thanks for preliminary analisys
There is why amount of committed blocks decreses with time.
I don't think it is expected behaviour just because:
- After restart it returns to initial performance
- Performance degradation is dramatical. After few weeks after launch it decreases from 2 blocks per second to 1 block per 30 seconds.
Performance degradation is dramatical. After few weeks after launch it decreases from 2 blocks per second to 1 block per 30 seconds.
Have you checked it or it's just a math ?
Have you checked it or it's just a math ?
It is results of our experiments with deployment from early November. We see it with our custom smart contracts as well as with exonum_cryptocurrency_advanced example.
Here is another two-day test.
Deployment: local
Binary: exonum_cryptocurrency_advanced
Number of nodes: 2
Configuration: default
@aleksuss Do you have any ideas or plans on this? Do you think it is a problem?
Yes. It seems that this is a problem. To be honest, the project is on hold now and I can't promise that the issue will be fixed ASAP. But I will keep in mind the problem and when I will have a free time I'll try to investigate the issue more accurate.
@aleksuss do you have any idea on what could be the cause? This is affecting us and we are willing to help/investigate and maybe open a pull request to solve it.
Which kind of the database do you use in your tests ? TemporaryDB or RocksDB ?
We always use RocksSB
We're also using RocksDB
I did a test with the TemporaryDB... It takes 30 mins for the CPU to reach 100% on my quad core macbook pro. Running a single node and just standing idle.
Yes. It's expected behaviour. 'TemporaryDB` is for running tests only.
It's expected behaviour to run the app under 100% of CPU all the time? Do you know the reason why it is so eager on CPU like this?
Another question: the application is running with 16 actix-rt workers. What is the reasoning for that? I'd say 16 for a quad core CPU is a bit too much. And I'm not counting the the other actix-server threads.
I mean when you're using TemporaryDB. There is a BTreeMap under the hood of TemporaryDB. When a caller invoke a snapshot method then created a copy of whole collection. It's expensive operation with huge amount of data inside.
I understand that it should be heavier than rocks-db. I'm trying to understand what is happening and why exonum is being so heavy on CPU. When using rocks-db the situation is not much better, it just takes more time to happen.
I expect an application standing idle to use a "constant" amount of resources, instead the application is always taking more CPU.
And again: where can be configured how many threads the application will start?
Amount of threads for starting is not configurable option. It depends on your CPU.
I expect an application standing idle to use a "constant" amount of resources
The idle term is not appropriate for a blockchain software because it always writes data into a database by creating new blocks.
I know. But from what you're saying I'm understanding that at some point the application will be unusable because the CPU consumption just grows and this is fine and expected.
We have investigated the issue using callgrind. Attached are three callgrind outputs obtained from running a plain node without executing transactions. The callgrind dump was always triggered after handle_majority_precommits, i.e. after each block.
Exonum was built using RUSTFLAGS="-Cforce-frame-pointers=yes" cargo build --release with the following added to Cargo.toml for proper debugging symbols:
[profile.release]
debug = true
Afterwards, the callgrind command line was:
valgrind --tool=callgrind --collect-jumps=yes --separate-threads=yes --dump-after='*handle_majority_precommits*' ./target/release/ledger run --node-config node.toml --db-path db --master-key-pass pass:""
callgrind.out.80761.10-01: snapshot after block 10
callgrind.out.80761.601-01: snapshot after block 601
...now the node was restarted...
callgrind.out.80915.603-01: snapshot after block 603 on the restarted node
The time spent in handle_majority_precommits was ~14 ms for block 10, and already ~40 ms for block 601. After restarting the node, it was down to ~15 ms again for block 603.
You can investigate the files with KCacheGrind or QCacheGrind.
It might be worth comparing especially the dumps for block 601 (before restart) and 603 (after restart), where it appears that the issue could be with the iterators on RocksDb. Those seem to take much longer and longer with each block. [1]
These iterators appear to be called through exonum::merkledb::backends::rocksdb::RocksDbSnapshot::iter().
I have issues following the control flow in cachegrind, but naively it seems be called somewhere like this:
exonum::runtime::dispatcher::Dispatcher::commit_block()
\___
+- exonum::runtime::dispatcher::schema::Schema<exonum::merkledb::db::Fork>::activate_pending()
+- exonum::runtime::dispatcher::schema::Schema<exonum::merkledb::db::Fork>::take_modified_instances()
+- exonum::runtime::dispatcher::schema::Schema<exonum::merkledb::db::Fork>::take_pending_artifacts()
\_____ exonum::merkledb::views::View::iter_bytes()
\_____ exonum::merkledb::backends::rocksdb::RocksDbSnapshot::rocksdb_iter()
exonum::runtime::dispatcher::Dispatcher::before_transactions()
\___ exonum::merkledb::Patch::iter()
\_____ exonum::merkledb::backends::rocksdb::RocksDbSnapshot::rocksdb_iter()
According to the link [1] below this could be relieved by replacing self.snapshot.iterator_cf with self.snapshot.iterator_cf_opt and passing additional options such as iterate_upper_bound, but I don't know enough about Exonum to propose a fix yet.
I hope this can help shed some light on the issue and can run more of these tests if need be.
[1] Consider this blog post about the performance of RocksDb iterators in the presence of tombstones and how to improve it.
Thank you for this report. I tried to find out the reason of performance regression by profiling and I've got the same result.
Now I'm trying to understand what has been changed and how to fix it. Because I have a service on exonum 0.12.2 which started 9 months ago and it works fine. There is no such problem. Maybe a turning of the rocksdb settings could help to fix the issue.
UPD. At this time I've found out that a reason of such behaviour is a continuously growing size of WAL journal of the rocksdb. A manually decreasing a size of WAL journal here seems fixes the issue:
defaults.set_max_total_wal_size(1 << 20);I'm trying to find out how to configure WAL journal correctly and will provide more details and a patch soon.
Hi guys! Are there any updates?
I asked about this issue on https://github.com/facebook/rocksdb but I haven't received any answer yet. What I can propose right now is to add a property to a config by which you could to set a max_total_wal_size. E.g. I've checked that a limit to 20mb fixes this issue.
Sorry, I haven't gotten around to wrapping up our investigation on this, yet...
We have ran the example with a limit on the max_total_wal_size for several days and that does fix this issue. However, it causes the number of open files to permanently increase, until the limit of the OS (run ulimit -n) is reached.
But, the good news is that the combination of limiting max_total_wal_size and defaults.set_max_open_files seems to resolve both issues!
So yes, exposing max_total_wal_size (max_open_files is already configurable) would definitely help.
PR #2070 fixes this issue
Wow this thing is alive
Related issue.