[ Feature request ] An event for progress?
davehayes opened this issue · comments
- Minion version: 10.08
- Perl version: 5.32
- Operating system: FreeBSD 11.3
So you provide this nice thing called 'progress' in notes. That's great and it works. However, on my first (always naive) attempt to use Minion I find myself polling notes periodically because there's no event to subscribe to.
Is it possible to fire an event when a job updates notes (or notes->progress)? Failing that, is it possible to document how to use progress to actually deliver progress data in an event-driven way?
Apologies if I missed anything in the documentation that covers this. Thanks in advance.
Theoretically this would be possible to implement with PostgreSQL listen/notify, but the SQLite and MySQL backends could not provide the feature. So we are not going to do it. And it's too complex to include as a documentation example.
Ok, so could you at least mention in the documentation that using progress requires polling the database?
How about you point out specifically the sections of the documentation that you think are too ambiguously worded? And then we'll see about rephrasing.
Well I'm not sure it's an issue of ambiguity as much as omission. You mention progress at first in the Minion docs here:
Minion is a high performance job queue for the Perl programming
language, with support for multiple named queues, priorities, delayed
jobs, job dependencies, job progress,...
Then you hint at the idea here further down in the same page:
job
my $job = $minion->job($id);
Get Minion::Job object without making any changes to the actual job or
return "undef" if job does not exist.
# Check job state
my $state = $minion->job($id)->info->{state};
# Get job metadata
my $progress = $minion->$job($id)->info->{notes}{progress};
I guess that can be generalized as metadata but as far as I can tell there's really two metadatum ... notes and results. Progress is really a subset of notes.
The word progress is not mentioned further. Digging into the documentation, there appears to be confusion in Minion::Job's docs because in the same page you first say this:
info
my $info = $job->info;
Get job information.
# Check job state
my $state = $job->info->{state};
# Get job metadata
my $progress = $job->info->{notes}{progress};
but further down you say this:
note
my $bool = $job->note(mojo => 'rocks', minion => 'too');
Change one or more metadata fields for this job. ...
# Share progress information
$job->note(progress => 95);
Perhaps I'm mistaken, but I really see 'note' as the metadata since it's just an arbitrary hash reference conceptually.
Nowhere else that I have found do I see information or examples as to how to retrieve this progress information when it changes. Maybe you could add some blurb that says "getting the notes requires polling the database for them so you can detect when they change"? I just threw that out there. ;)
Minion explicitly states that it supports job progress, so it's natural to me to go looking for where that support is.
Yes, you're reading too much into the example. The notes feature is job progress information, there is no subset.
I think what it all comes down to is that the Minion guide explaining all the features with examples is still missing.
The guide is now a planned feature, but i have no idea when i'll get around to actually writing it. #100
A guide which explains all the features with examples would address my concerns. IMHO if I have to look at your code to understand what something actually does, the documentation is by definition lacking in that area.
Not that I don't like looking at your code either, you have a uniquely terse style that can be fun to figure out...and not that people who are using your code shouldn't look at it anyway...but this is strictly a measure of documentation quality.
I've addressed the event issue myself because I use the Pg backend. What would it take to get support for listen/notify in SQLite or MySQL? Can you do listen/notify without Pg? I'm just curious here.
It's not possible with SQLite because there is no server.
It's not possible with SQLite because there is no server.
I'm just covering all the bases here. I recognize this, but maybe someone has written something that does pubsub for SQLite or even for nothing at all (rabbiveesh's comment ... redis for example). I'm trying to work within the bounds of what has been stated to get the feature I feel should likely be a part of Minion.
So to my original request ... not a wanted feature then? I can live with that as long as every other option has been exhausted. Maybe you could make pubsub event an optional feature of Minion::Backend::Pg?
You're working with too blunt of a hammer. You're assuming that EVERY item
put in notes is related to progress, which isn't necessarily true.
You might be misunderstanding me. I see this:
# Get job metadata
my $progress = $job->info->{notes}{progress};
and I assume progress is a subset of notes. I even said that:
I guess that can be generalized as metadata but as far as I can tell there's really two metadatum ... notes and results. Progress is really a subset of notes
Not sure what you are perceiving here. :)
So if notes is a hashref and progress can be a hashref (it always is in my case) and progress/hashref is a key/value pair in notes... in that case is it not valid to say that progress is a subset of notes?
You might want to continue that discussion in the Google Group or on IRC.