Callback after a batch is flushed?

[Francis Lavoie]

We have a primary-replica setup for our database, and for safety in our cron jobs, we like to check the replica delay to make sure we're not adding too much pressure.

It would be nice if we could register a callback to the BulkDeleter (or BulkOperator in general) which is called after each batch is flushed so we can self-throttle.

Currently we'd need to check ->getPendingOperations() == 0 at the end of the ->queue() loop to detect if we just flushed, which is fine, but somewhat janky. I think a callback approach would be cleaner.

I'm thinking an API like this:

$deleter = new BulkDeleter($pdo, 'table', [...$cols]);
$deleter->setFlushCallback(fn() => custom_replica_delay_check());
foreach ($toDelete as $entry) {
	$deleter->queue($entry);
}
$deleter->flush();

[Benjamin Morel]

Hi @francislavoie!

Did you notice that queue() returns true when a batch has been flushed? It's to be used specifically for this purpose.

That being said, I'm not against replacing it with a callback system, which may be a cleaner API indeed. Should it be called only when queue() flushes, or when calling flush() explicitly as well?

[Francis Lavoie]

Did you notice that queue() returns true when a batch has been flushed?

Ooh, I did miss that. Oops!

Should it be called only when queue() flushes, or when calling flush() explicitly as well?

I think both, but we could pass a boolean to the callback to mean "is last" or whatever.

But then again there's no guarantee that flush() is actually final because there could be a legit reason to manually flush in the middle of the loop (I can't think of an example why but I imagine someone could have a reason).

Hmm, tricky-tricky.

Anyway I don't feel the need as strongly for this feature now that I know there's that return book (and to be honest I figured out I could use ->getPendingOperations() == 0 part-way through writing the issue 😅) so we can probably close this and call it "documentation in case someone else comes along" 🤷‍♂️

Thanks for your excellent libraries once again, we use many of them ☺️

[Benjamin Morel]

I quite like the callback approach anyway, I'll keep it in mind.

In the meantime, don't hesitate to submit a PR to improve the README!

Thanks for your excellent libraries once again, we use many of them ☺️

Much appreciated 🙂