algorand / conduit

Algorand's data pipeline framework.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Delegate Importer Retries to Conduit

tzaffi opened this issue · comments

Remove retry logic in algod importer

Now that conduit.yml provides a pipeline retry-count: configuration, plugins generally ought not implement their own retry logic.

Your environment

Local mac

Steps to reproduce

  1. set retry-count: 1
  2. set importer.name: algod
  3. start a fresh follower node against mainnet
  4. Try starting with a round that isn't cached on the relays. EG: conduit -d ./data -r 10009217
  5. We know that this is bound to fail because only the most recent 365 catchpoints (up to 3.65 M rounds from HEAD) are stored on the relays

Expected behaviour

After 1 retry conduit immediately reports back an error and exist.

Actual behaviour

It appears to be trying 5 times, because the algod importer is:

{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 0)","time":"2023-05-03T22:16:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 1)","time":"2023-05-03T22:17:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 2)","time":"2023-05-03T22:18:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 3)","time":"2023-05-03T22:19:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 4)","time":"2023-05-03T22:20:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2023-05-03T22:20:11-05:00"}
{"__type":"Conduit","_name":"main","level":"error","msg":"failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2023-05-03T22:20:11-05:00"}
{"__type":"Conduit","_name":"main","level":"info","msg":"Pipeline round: 10009217","time":"2023-05-03T22:20:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 0)","time":"2023-05-03T22:21:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 1)","time":"2023-05-03T22:22:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 2)","time":"2023-05-03T22:23:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 3)","time":"2023-05-03T22:24:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 4)","time":"2023-05-03T22:25:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2023-05-03T22:25:12-05:00"}
{"__type":"Conduit","_name":"main","level":"error","msg":"failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2023-05-03T22:25:12-05:00"}
{"__type":"Conduit","_name":"main","level":"error","msg":"Pipeline has exceeded maximum retry count (1) - stopping...","time":"2023-05-03T22:25:12-05:00"}
{"__type":"processor","_name":"file_writer","level":"info","msg":"latest round on file: 10009217","time":"2023-05-03T22:25:12-05:00"}

Exiting with error:
failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {"message":"failed to retrieve information from the ledger"}