Batch enqueue

[Sebastian Riedel]

This came up in the Matrix channel. Apparently a few of our users would like to see support for batch enqueueing multiple jobs at once. Like in a database transaction. In this issue you can collect use cases to work out the exact requirements and maybe even the correct API.

But be aware that this has to work for all the databases and programming languages Minion supports.

[Pedro Melo]

Hi,

I'll split the requirements into two items. I think this ticket is only for the first one.

1. allow users to create multiple jobs in a single database transaction: either all of them are created or none are;
2. allow Minion database access to be done inside an external-controlled transaction: this would allow users to (for example) get an API call to update our DB tables and then create a job to do some post-processing, all inside a single transaction. This assumes of course that the Minion tables share the main database.

I'll focus on the first item here.

The simplest API I can think off is a enqueue_batch() that takes a list of arrayRefs of parameters to the enqueue() API. Something like this would be enough:

my $arrayRefOfJobIds = $minion->batch_enqueue(
  [ ... enqueue parameters ... ],
  [ ... enqueue parameters ... ],
  [ ... enqueue parameters ... ],
  ...
);

An alternative is similar to karjala@matrix suggestion. We would take a codeRef and all calls to $minion->enqueue() inside that callback would be inside a single transaction. Like this:

$minion->tx(sub ($minion_tx, $db_tx) {
  # do other work on DB
  $db_tx->insert(...);

  # and queue multiple jobs
  $minion_tx->enqueue(....);
  $minion_tx->enqueue(....);
  $minion_tx->enqueue(....);
  ...
});

The advantage of this is that you can use the $db_tx handle to do other DB operations all of them inside the same transaction.

I think either of these is an options and both seem to be compatible with the three languages you mention.

Best regards.

Bye,

[Alexander Karelas]

I'm covered by @melo's comments.

For me it's very important that I am able to execute arbitrary SQL statements to my database, in the same transaction that enqueues minion jobs.

My use case is that I have to deduct credits from a user's account for adding a minion job, so these 2 things have to happen on the same transaction.

Thanks.