[BUG] - Missing prometheus metrics in cardano-node:8.11.0-sancho

[Michal Štefanec]

External

Area
Other Any other topic (Delegation, Ranking, ...).

Summary
In the docker version cardano-node:8.11.0-sancho I'm missing the Prometheus metrics about the Cardano chain (e.g. cardano_node_metrics_blockNum_int). Metrics about peerSelection are present.

Expected behavior
A clear and concise description of what you expected to happen.

System info (please complete the following information):

• Node version: cardano-node:8.11.0-sancho

Screenshots and attachments

• If applicable, add screenshots, config files and/or logs to help explain the problem.

[Michal Štefanec]

All metrics I got are only these

bash-5.2# curl localhost:12798/metrics
cardano_node_metrics_connectionManager_incomingConns 0
cardano_node_metrics_peerSelection_WarmNonRootPeersPromotions 0
cardano_node_metrics_peerSelection_ActiveBootstrapPeers 0
cardano_node_metrics_RTS_gcLiveBytes_int 22901736
cardano_node_metrics_peerSelection_KnownBigLedgerPeers 0
cardano_node_metrics_peerSelection_WarmNonRootPeersDemotions 0
rts_gc_current_bytes_used 22901736
rts_gc_num_bytes_usage_samples 7
rts_gc_gc_cpu_ms 9299
cardano_node_metrics_peerSelection_ActivePeers 0
cardano_node_metrics_connectionManager_prunableConns 0
cardano_node_metrics_peerSelection_cold 6
cardano_node_metrics_peerSelection_WarmPeersDemotions 0
cardano_node_metrics_peerSelection_EstablishedPeers 3
cardano_node_metrics_peerSelection_ActivePeersDemotions 0
cardano_node_metrics_peerSelection_ColdBigLedgerPeersPromotions 0
rts_gc_cumulative_bytes_used 48941576
cardano_node_metrics_peerSelection_WarmBootstrapPeersDemotions 0
cardano_node_metrics_connectionManager_unidirectionalConns 0
cardano_node_metrics_peerSelection_ActiveLocalRootPeers 0
rts_gc_mutator_wall_ms 167285
rts_gc_cpu_ms 70817
rts_gc_num_gcs 1756
cardano_node_metrics_nodeStartTime_int 1716365285
cardano_node_metrics_peerSelection_ActiveNonRootPeers 0
cardano_node_metrics_Stat_threads_int 16
cardano_node_metrics_Mem_resident_int 156725248
ekg_server_timestamp_ms 1716365454295
cardano_node_metrics_peerSelection_EstablishedLocalRootPeers 0
cardano_node_metrics_RTS_mutticks_int 6151
rts_gc_wall_ms 170208
cardano_node_metrics_peerSelection_ColdNonRootPeersPromotions 0
cardano_node_metrics_peerSelection_ActiveBigLedgerPeers 0
cardano_node_metrics_peerSelection_ActiveBigLedgerPeersDemotions 0
cardano_node_metrics_inboundGovernor_idle 0
cardano_node_metrics_peerSelection_EstablishedNonRootPeers 0
cardano_node_metrics_peerSelection_ActiveLocalRootPeersDemotions 0
cardano_node_metrics_peerSelection_coldBigLedgerPeers 0
cardano_node_metrics_inboundGovernor_hot 0
rts_gc_par_max_bytes_copied 536241168
cardano_node_metrics_peerSelection_WarmPeersPromotions 0
cardano_node_metrics_peerSelection_KnownPeers 9
rts_gc_par_avg_bytes_copied 565363896
cardano_node_metrics_peerSelection_ColdBootstrapPeersPromotions 0
rts_gc_max_bytes_slop 409992
rts_gc_init_cpu_ms 4
cardano_node_metrics_peerSelection_KnownBootstrapPeers 9
cardano_node_metrics_RTS_gcMajorNum_int 7
rts_gc_par_tot_bytes_copied 565363896
cardano_node_metrics_peerSelection_ActiveBootstrapPeersDemotions 0
rts_gc_bytes_allocated 99056310280
rts_gc_current_bytes_slop 424984
rts_gc_bytes_copied 565363896
rts_gc_init_wall_ms 3
rts_gc_max_bytes_used 16815336
cardano_node_metrics_connectionManager_outgoingConns 3
cardano_node_metrics_inboundGovernor_cold 3
cardano_node_metrics_peerSelection_WarmBootstrapPeersPromotions 0
cardano_node_metrics_peerSelection_KnownNonRootPeers 0
cardano_node_metrics_peerSelection_warm 3
cardano_node_metrics_RTS_gcMinorNum_int 1749
rts_gc_peak_megabytes_allocated 100
cardano_node_metrics_Stat_cputicks_int 7082
rts_gc_mutator_cpu_ms 61519
forging_enabled 1
cardano_node_metrics_peerSelection_EstablishedBigLedgerPeers 0
cardano_node_metrics_RTS_gcticks_int 929
cardano_node_metrics_peerSelection_ActiveNonRootPeersDemotions 0
cardano_node_metrics_connectionManager_duplexConns 3
cardano_node_metrics_RTS_gcHeapBytes_int 104857600
cardano_node_metrics_peerSelection_EstablishedBootstrapPeers 3
cardano_node_metrics_peerSelection_KnownLocalRootPeers 0
cardano_node_metrics_peerSelection_RootPeers 9
cardano_node_metrics_peerSelection_hot 0
cardano_node_metrics_peerSelection_WarmLocalRootPeersPromotions 0
cardano_node_metrics_peerSelection_warmBigLedgerPeers 0
cardano_node_metrics_peerSelection_WarmBigLedgerPeersPromotions 0
cardano_node_metrics_peerSelection_WarmBigLedgerPeersDemotions 0
cardano_node_metrics_peerSelection_hotBigLedgerPeers 0
rts_gc_gc_wall_ms 2924
cardano_node_metrics_inboundGovernor_warm 0
cardano_node_metrics_peerSelection_ColdPeersPromotions 0

[Michal Štefanec]

My node config file

  "AlonzoGenesisFile": "sanchonet-alonzo-genesis.json",
  "AlonzoGenesisHash": "8bedcaea62107d8a79ed5293b0027b3f8706a4bc2422f33380cb1fd01c6fa6ec",
  "ByronGenesisFile": "sanchonet-byron-genesis.json",
  "ByronGenesisHash": "785eb88427e136378a15b0a152a8bfbeec7a611529ccda29c43a1e60ffb48eaa",
  "ConwayGenesisFile": "sanchonet-conway-genesis.json",
  "ConwayGenesisHash": "79766b0c18076e9abefc806526714f30d42b14dba9f187fb316bd4e70c815ef4",
  "EnableP2P": true,
  "ExperimentalHardForksEnabled": true,
  "ExperimentalProtocolsEnabled": true,
  "LastKnownBlockVersion-Alt": 0,
  "LastKnownBlockVersion-Major": 3,
  "LastKnownBlockVersion-Minor": 1,
  "MinNodeVersion": "8.11.0",
  "PeerSharing": false,
  "Protocol": "Cardano",
  "RequiresNetworkMagic": "RequiresMagic",
  "ShelleyGenesisFile": "sanchonet-shelley-genesis.json",
  "ShelleyGenesisHash": "f94457ec45a0c6773057a529533cf7ccf746cb44dabd56ae970e1dbfb55bfdb2",
  "TargetNumberOfActivePeers": 20,
  "TargetNumberOfEstablishedPeers": 50,
  "TargetNumberOfKnownPeers": 100,
  "TargetNumberOfRootPeers": 100,
  "TestAllegraHardForkAtEpoch": 0,
  "TestAlonzoHardForkAtEpoch": 0,
  "TestMaryHardForkAtEpoch": 0,
  "TestShelleyHardForkAtEpoch": 0,
  "TraceAcceptPolicy": true,
  "TraceBlockFetchClient": false,
  "TraceBlockFetchDecisions": false,
  "TraceBlockFetchProtocol": false,
  "TraceBlockFetchProtocolSerialised": false,
  "TraceBlockFetchServer": false,
  "TraceChainDb": true,
  "TraceChainSyncBlockServer": false,
  "TraceChainSyncClient": false,
  "TraceChainSyncHeaderServer": false,
  "TraceChainSyncProtocol": false,
  "TraceConnectionManager": true,
  "TraceDNSResolver": true,
  "TraceDNSSubscription": true,
  "TraceDiffusionInitialization": true,
  "TraceErrorPolicy": true,
  "TraceForge": true,
  "TraceHandshake": true,
  "TraceInboundGovernor": true,
  "TraceIpSubscription": true,
  "TraceLedgerPeers": true,
  "TraceLocalChainSyncProtocol": false,
  "TraceLocalConnectionManager": true,
  "TraceLocalErrorPolicy": true,
  "TraceLocalHandshake": true,
  "TraceLocalRootPeers": true,
  "TraceLocalTxSubmissionProtocol": false,
  "TraceLocalTxSubmissionServer": false,
  "TraceMempool": true,
  "TraceMux": false,
  "TracePeerSelection": true,
  "TracePeerSelectionActions": true,
  "TracePublicRootPeers": true,
  "TraceServer": true,
  "TraceTxInbound": false,
  "TraceTxOutbound": false,
  "TraceTxSubmissionProtocol": false,
  "TracingVerbosity": "NormalVerbosity",
  "TurnOnLogMetrics": false,
  "TurnOnLogging": true,
  "defaultBackends": [
    "KatipBK"
  ],
  "defaultScribes": [
    [
      "StdoutSK",
      "stdout"
    ]
  ],
  "hasEKG": 12788,
  "hasPrometheus": [
    "0.0.0.0",
    12798
  ],
  "minSeverity": "Info",
  "options": {
    "mapBackends": {
      "cardano.node.metrics": [
        "EKGViewBK"
      ],
      "cardano.node.resources": [
        "EKGViewBK"
      ]
    },
    "mapSubtrace": {
      "cardano.node.metrics": {
        "subtrace": "Neutral"
      }
    }
  },
  "rotation": {
    "rpKeepFilesNum": 10,
    "rpLogLimitBytes": 5000000,
    "rpMaxAgeHours": 24
  },
  "setupBackends": [
    "KatipBK"
  ],
  "setupScribes": [
    {
      "scFormat": "ScText",
      "scKind": "StdoutSK",
      "scName": "stdout",
      "scRotation": null
    }
  ]
}

[Samuel Leathers]

Did you truncate after the respin? https://discord.com/channels/826816523368005654/1141734250438197370/1240827313671766051

Your metrics show that you have no peers that can give you a block, so it's likely you're either on a different fork and need to rollback the chain/replay or there's something wrong with your topology file you're using.

[Michal Štefanec]

That's the case, thank you!