descholar-ceo/mosquito

Mosquito is a generic background job runner written primarily for Crystal. Significant inspiration from experience with the successes and failings many Ruby gems in this vein.

Mosquito currently provides these features:

Delayed execution
Scheduled / Periodic execution
Job Storage in Redis
Automatic rescheduling of failed jobs
Progressively increasing delay of failed jobs
Dead letter queue of jobs which have failed too many times
Rate limited jobs

Current Limitations:

Visibility into a running job network and queue is limited. There is a working proof of concept visualization API and bare-bones terminal application.

Project State

The Mosquito project is stable. A few folks are using Mosquito in production, and it's going okay.

There are some features which would be nice to have, but what is here is both tried and tested.

If you're using Mosquito, please get in touch on the Discussion board or on Crystal chat with any questions, feature suggestions, or feedback.

Installation

Update your shard.yml to include mosquito:

dependencies:
+  mosquito:
+    github: mosquito-cr/mosquito

Usage

Step 1: Define a queued job

# src/jobs/puts_job.cr
class PutsJob < Mosquito::QueuedJob
  param message : String

  def perform
    puts message
  end
end

Step 2: Trigger that job

# src/<somewher>/<somefile>.cr
PutsJob.new(message: "ohai background job").enqueue

Step 3: Run your worker to process the job

# src/worker.cr

Mosquito.configure do |settings|
  settings.redis_url = ENV["REDIS_URL"]
end

Mosquito::Runner.start

crystal run src/worker.cr

Success

> crystal run src/worker.cr
2017-11-06 17:07:29 - Mosquito is buzzing...
2017-11-06 17:07:51 - Running task puts_job<...> from puts_job
2017-11-06 17:07:51 - [PutsJob] ohai background job
2017-11-06 17:07:51 - task puts_job<...> succeeded, took 0.0 seconds

More information about queued jobs in the manual.

Periodic Jobs

Periodic jobs run according to a predefined period -- once an hour, etc.

This periodic job:

class PeriodicallyPutsJob < Mosquito::PeriodicJob
  run_every 1.minute

  def perform
    emotions = %w{happy sad angry optimistic political skeptical epuhoric}
    puts "The time is now #{Time.local} and the wizard is feeling #{emotions.sample}"
  end
end

Would produce this output:

2017-11-06 17:20:13 - Mosquito is buzzing...
2017-11-06 17:20:13 - Queues: periodically_puts_job
2017-11-06 17:20:13 - Running task periodically_puts_job<...> from periodically_puts_job
2017-11-06 17:20:13 - [PeriodicallyPutsJob] The time is now 2017-11-06 17:20:13 and the wizard is feeling skeptical
2017-11-06 17:20:13 - task periodically_puts_job<...> succeeded, took 0.0 seconds
2017-11-06 17:21:14 - Queues: periodically_puts_job
2017-11-06 17:21:14 - Running task periodically_puts_job<...> from periodically_puts_job
2017-11-06 17:21:14 - [PeriodicallyPutsJob] The time is now 2017-11-06 17:21:14 and the wizard is feeling optimistic
2017-11-06 17:21:14 - task periodically_puts_job<...> succeeded, took 0.0 seconds
2017-11-06 17:22:15 - Queues: periodically_puts_job
2017-11-06 17:22:15 - Running task periodically_puts_job<...> from periodically_puts_job
2017-11-06 17:22:15 - [PeriodicallyPutsJob] The time is now 2017-11-06 17:22:15 and the wizard is feeling political
2017-11-06 17:22:15 - task periodically_puts_job<...> succeeded, took 0.0 seconds

More information on periodic jobs in the manual.

Advanced usage

For more advanced topics, including use with Lucky Framework, throttling or rate limiting, check out the full manual.

Contributing

Contributions are welcome. Please fork the repository, commit changes on a branch, and then open a pull request.

Crystal Versions

Mosquito aims to be compatible with the latest Crystal release, and the latest patch for all post-1.0 minor crystal versions.

For development purposes you're encouraged to stay in sync with .tool-versions.

Testing

This repository uses minitest for testing. As a result, crystal spec doesn't do anything helpful. Do this instead:

make test

In lieu of crystal spec bells and whistles, Minitest provides a nice alternative to running one test at a time instead of the whole suite.

descholar-ceo / mosquito