node stops while syncing - external-node-1 exited with code 137
shaky opened this issue · comments
hi there,
I'm trying to run the node announced here #526 but seems like it stops running after approximately 10 mins.
docker compose --file mainnet-external-node-docker-compose.yml up
WARN[0000] /Users/censored/Projects/zksync-era/docs/guides/external-node/docker-compose-examples/mainnet-external-node-docker-compose.yml: `version` is obsolete
[+] Running 4/0
✔ Container docker-compose-examples-postgres-1 Running 0.0s
✔ Container docker-compose-examples-prometheus-1 Running 0.0s
✔ Container docker-compose-examples-grafana-1 Running 0.0s
✔ Container docker-compose-examples-external-node-1 Created 0.0s
Attaching to external-node-1, grafana-1, postgres-1, prometheus-1
external-node-1 | 2024-05-13T18:10:04.413731Z INFO zksync_external_node: No sentry URL was provided
external-node-1 | 2024-05-13T18:10:05.657965Z INFO zksync_external_node::metrics: Metadata for rustc that this EN was compiled with: RustcMetadata { version: "1.74.0-nightly", commit_hash: Some("5c6a7e71cd66705c31c9af94077901a220f0870c"), commit_date: Some("2023-08-20"), channel: "nightly", host: "aarch64-unknown-linux-gnu", llvm: Some("17.0") }
external-node-1 | 2024-05-13T18:10:05.658057Z INFO zksync_external_node::metrics: Setting general node information: ExternalNodeInfo { server_version: "23.0.0", l1_chain_id: 1, l2_chain_id: 324, postgres_pool_size: 10 }
external-node-1 | 2024-05-13T18:10:05.681284Z INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { max_size: 10, acquire_timeout: 30s, statement_timeout: None }
external-node-1 | 2024-05-13T18:10:05.681419Z INFO zksync_external_node: Main node URL is: https://zksync2-mainnet.zksync.io:443/
external-node-1 | 2024-05-13T18:10:05.682213Z WARN zksync_external_node: The external node is in the alpha phase, and should be used with caution.
external-node-1 | 2024-05-13T18:10:05.682235Z INFO zksync_external_node: Started the external node
external-node-1 | 2024-05-13T18:10:05.697082Z INFO zksync_db_connection::connection_pool: Created DB pool with parameters ConnectionPoolBuilder { max_size: 1, acquire_timeout: 30s, statement_timeout: None }
external-node-1 | 2024-05-13T18:10:05.697245Z INFO zksync_external_node::version_sync_task: Starting syncing protocol version of blocks
external-node-1 | 2024-05-13T18:10:05.709619Z INFO zksync_external_node::init: Node has no genesis L1 batch and snapshot recovery information: SnapshotRecoveryStatus { l1_batch_number: L1BatchNumber(480164), l1_batch_root_hash: 0xccf9d6c8fd8dbf965da4e366b69bf299820c1c469ddc4090af76a77cebd1ec87, l1_batch_timestamp: 1715556481, miniblock_number: MiniblockNumber(33835879), miniblock_hash: 0x0ca9fc8ca1c6e7834a9aee185686f1cee9689aa1be5f080cf4d6b4e6f3e6c8fc, miniblock_timestamp: 1715557231, protocol_version: Version22, storage_logs_chunks_processed: [true, false, true, false, true, true, true, true, true, false, true, true, true, false, false, true, false, true, false, true, true, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false] }
external-node-1 | 2024-05-13T18:10:05.710090Z INFO zksync_external_node::init: Chosen node initialization strategy: SnapshotRecovery
external-node-1 | 2024-05-13T18:10:05.710094Z WARN zksync_external_node::init: Proceeding with snapshot recovery. This is an experimental feature; use at your own risk
external-node-1 | 2024-05-13T18:10:05.740030Z INFO zksync_snapshots_applier: Re-initialized snapshots applier after reset/failure in 2.520125ms
external-node-1 | 2024-05-13T18:10:05.741034Z INFO zksync_snapshots_applier: Processing storage logs chunk 1
external-node-1 | 2024-05-13T18:10:05.741805Z INFO zksync_snapshots_applier: Processing storage logs chunk 3
external-node-1 | 2024-05-13T18:10:05.741868Z INFO zksync_snapshots_applier: Processing storage logs chunk 9
external-node-1 | 2024-05-13T18:10:05.741924Z INFO zksync_snapshots_applier: Processing storage logs chunk 13
external-node-1 | 2024-05-13T18:10:05.741976Z INFO zksync_snapshots_applier: Processing storage logs chunk 14
external-node-1 | 2024-05-13T18:10:05.742033Z INFO zksync_snapshots_applier: Processing storage logs chunk 16
external-node-1 | 2024-05-13T18:10:05.742077Z INFO zksync_snapshots_applier: Processing storage logs chunk 18
external-node-1 | 2024-05-13T18:10:05.742108Z INFO zksync_snapshots_applier: Processing storage logs chunk 21
external-node-1 | 2024-05-13T18:10:05.742193Z INFO zksync_snapshots_applier: Processing storage logs chunk 22
external-node-1 | 2024-05-13T18:10:05.742425Z INFO zksync_snapshots_applier: Processing storage logs chunk 23
external-node-1 | 2024-05-13T18:11:07.336709Z INFO zksync_snapshots_applier: Loaded 996982 storage logs from GCS for chunk 9 in 61.59648432s
external-node-1 | 2024-05-13T18:11:07.337231Z INFO zksync_snapshots_applier: Loading 996982 storage logs into Postgres
external-node-1 | 2024-05-13T18:11:11.106609Z INFO zksync_snapshots_applier: Loaded 998258 storage logs from GCS for chunk 22 in 65.366064822s
external-node-1 | 2024-05-13T18:11:11.107189Z INFO zksync_snapshots_applier: Loading 998258 storage logs into Postgres
external-node-1 | 2024-05-13T18:11:14.320519Z INFO zksync_snapshots_applier: Loaded 998477 storage logs from GCS for chunk 13 in 68.58024799s
external-node-1 | 2024-05-13T18:11:14.336580Z INFO zksync_snapshots_applier: Loading 998477 storage logs into Postgres
external-node-1 | 2024-05-13T18:11:17.983291Z INFO zksync_snapshots_applier: Loaded 999664 storage logs from GCS for chunk 21 in 72.242830284s
external-node-1 | 2024-05-13T18:11:17.988663Z INFO zksync_snapshots_applier: Loading 999664 storage logs into Postgres
external-node-1 | 2024-05-13T18:11:20.573685Z INFO zksync_snapshots_applier: Loaded 998916 storage logs from GCS for chunk 14 in 74.832843118s
external-node-1 | 2024-05-13T18:11:20.581494Z INFO zksync_snapshots_applier: Loading 998916 storage logs into Postgres
external-node-1 | 2024-05-13T18:11:24.758088Z INFO zksync_snapshots_applier: Loaded 1000153 storage logs from GCS for chunk 3 in 79.01740487s
external-node-1 | 2024-05-13T18:11:25.343647Z INFO zksync_snapshots_applier: Loaded 997139 storage logs from GCS for chunk 16 in 79.602744995s
external-node-1 | 2024-05-13T18:11:25.366844Z INFO zksync_snapshots_applier: Loading 1000153 storage logs into Postgres
external-node-1 | 2024-05-13T18:11:27.481143Z INFO zksync_snapshots_applier: Loading 997139 storage logs into Postgres
grafana-1 | logger=cleanup t=2024-05-13T18:11:28.14106067Z level=info msg="Completed cleanup jobs" duration=278.648042ms
postgres-1 | 2024-05-13 18:11:29.735 UTC [10157] ERROR: unexpected EOF on client connection with an open transaction
postgres-1 | 2024-05-13 18:11:29.735 UTC [10157] STATEMENT: COPY storage_logs(
postgres-1 | hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1 | created_at, updated_at
postgres-1 | )
postgres-1 | FROM STDIN WITH (DELIMITER '|')
postgres-1 | 2024-05-13 18:11:29.735 UTC [10156] LOG: incomplete message from client
postgres-1 | 2024-05-13 18:11:29.735 UTC [10156] STATEMENT: COPY storage_logs(
postgres-1 | hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1 | created_at, updated_at
postgres-1 | )
postgres-1 | FROM STDIN WITH (DELIMITER '|')
postgres-1 | 2024-05-13 18:11:29.736 UTC [10156] ERROR: unexpected EOF on client connection with an open transaction
postgres-1 | 2024-05-13 18:11:29.736 UTC [10156] STATEMENT: COPY storage_logs(
postgres-1 | hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1 | created_at, updated_at
postgres-1 | )
postgres-1 | FROM STDIN WITH (DELIMITER '|')
postgres-1 | 2024-05-13 18:11:29.735 UTC [10119] LOG: incomplete message from client
postgres-1 | 2024-05-13 18:11:29.735 UTC [10119] STATEMENT: COPY storage_logs(
postgres-1 | hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1 | created_at, updated_at
postgres-1 | )
postgres-1 | FROM STDIN WITH (DELIMITER '|')
postgres-1 | 2024-05-13 18:11:29.736 UTC [10119] ERROR: unexpected EOF on client connection with an open transaction
postgres-1 | 2024-05-13 18:11:29.736 UTC [10119] STATEMENT: COPY storage_logs(
postgres-1 | hashed_key, address, key, value, operation_number, tx_hash, miniblock_number,
postgres-1 | created_at, updated_at
postgres-1 | )
postgres-1 | FROM STDIN WITH (DELIMITER '|')
postgres-1 | 2024-05-13 18:11:29.737 UTC [10157] FATAL: terminating connection because protocol synchronization was lost
postgres-1 | 2024-05-13 18:11:29.737 UTC [10156] FATAL: terminating connection because protocol synchronization was lost
postgres-1 | 2024-05-13 18:11:29.740 UTC [10156] LOG: could not send data to client: Broken pipe
postgres-1 | 2024-05-13 18:11:29.740 UTC [10157] LOG: could not send data to client: Broken pipe
postgres-1 | 2024-05-13 18:11:29.753 UTC [10119] FATAL: terminating connection because protocol synchronization was lost
postgres-1 | 2024-05-13 18:11:29.753 UTC [10119] LOG: could not send data to client: Broken pipe
postgres-1 | 2024-05-13 18:11:29.768 UTC [9479] LOG: unexpected EOF on client connection with an open transaction
external-node-1 exited with code 137
postgres-1 | 2024-05-13 18:11:33.349 UTC [10092] LOG: unexpected EOF on client connection with an open transaction
postgres-1 | 2024-05-13 18:11:34.860 UTC [9481] LOG: unexpected EOF on client connection with an open transaction
postgres-1 | 2024-05-13 18:11:36.116 UTC [10065] LOG: unexpected EOF on client connection with an open transaction
Please advise. Thank you!
withdrawing this, I increased the RAM in docker, I'll observe and return in case of any other issues.