algorand / conduit

Algorand's data pipeline framework.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Pipelining Followups

tzaffi opened this issue · comments

Problem

Some issues remaining at the end of #128. However, the long validation process made it safer to push fixes as followups

Remove the *pImpl parameter from Retries() in conduit/pipeline/common.go

Per @winder

  • breaking out the context.Context out into a parameter, I was specifically looking for it to make sure the wait is properly terminated.
  • also breaking out RetryCount and logger. It's already a pure function in how it's used, but the signature doesn't guarantee that it's pure.

Alternatively, just make Retries private

Trim the special end of round message

In the final EC2 test, the size of the log at catchup grew from 14.1 GB to 16.2 GB. This is attributable to the change of the end of round log message. Currently it looks like

UPDATED Pipeline NextRound=31446285. FINISHED Pipeline round r=31446284 (6 txn) exported in 3.721356ms

In particular the UPDATED... prefix was added.

While this message requires a special format to comply with our testing (internal tools logstats.go and the E2E test), we can certainly trim it down and the UPDATED NextRound info is not crucial (it can be carved out as a DEBUG level message if not already present).

Urgency

Medium