Delegate Importer Retries to Conduit
tzaffi opened this issue · comments
Zeph Grunschlag commented
Remove retry logic in algod importer
Now that conduit.yml
provides a pipeline retry-count:
configuration, plugins generally ought not implement their own retry logic.
Your environment
Local mac
Steps to reproduce
- set
retry-count: 1
- set
importer.name: algod
- start a fresh follower node against
mainnet
- Try starting with a round that isn't cached on the relays. EG:
conduit -d ./data -r 10009217
- We know that this is bound to fail because only the most recent 365 catchpoints (up to 3.65 M rounds from HEAD) are stored on the relays
Expected behaviour
After 1 retry conduit immediately reports back an error and exist.
Actual behaviour
It appears to be trying 5 times, because the algod importer is:
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 0)","time":"2023-05-03T22:16:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 1)","time":"2023-05-03T22:17:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 2)","time":"2023-05-03T22:18:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 3)","time":"2023-05-03T22:19:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 4)","time":"2023-05-03T22:20:11-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2023-05-03T22:20:11-05:00"}
{"__type":"Conduit","_name":"main","level":"error","msg":"failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2023-05-03T22:20:11-05:00"}
{"__type":"Conduit","_name":"main","level":"info","msg":"Pipeline round: 10009217","time":"2023-05-03T22:20:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 0)","time":"2023-05-03T22:21:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 1)","time":"2023-05-03T22:22:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 2)","time":"2023-05-03T22:23:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 3)","time":"2023-05-03T22:24:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 4)","time":"2023-05-03T22:25:12-05:00"}
{"__type":"processor","_name":"algod","level":"error","msg":"failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2023-05-03T22:25:12-05:00"}
{"__type":"Conduit","_name":"main","level":"error","msg":"failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {\"message\":\"failed to retrieve information from the ledger\"}\n","time":"2023-05-03T22:25:12-05:00"}
{"__type":"Conduit","_name":"main","level":"error","msg":"Pipeline has exceeded maximum retry count (1) - stopping...","time":"2023-05-03T22:25:12-05:00"}
{"__type":"processor","_name":"file_writer","level":"info","msg":"latest round on file: 10009217","time":"2023-05-03T22:25:12-05:00"}
Exiting with error:
failed to get block for round 10009217 after 5 attempts, check node configuration: HTTP 404: {"message":"failed to retrieve information from the ledger"}