Giters

brandoncarl / taskco

TaskCo is a Redis-backed node.js library for creating background jobs, placing them on multiple queues, and processing them later.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

TaskCo should have graceful shutdown

brandoncarl opened this issue · comments

commented

Many PaaS providers (Heroku being a notable example), kill processes periodically. We need to respond to these deaths by re-queueing jobs.

TaskCo should not listen for these events, but rather, offer a shutdown method at each stage:

Company shutdown > factory shutdown > dispatcher shutdown > team shutdown > worker shutdown

commented

Currently in progress of working on this.

There have been two primary difficulties:

  1. BLPOP with no timeout makes it impossible to gracefully close down Redis connections.
  2. Difficulties shutting down code already running. This is necessary when a termination event is received.
commented

Nearly done with the following implementation:

  1. Affected Factories are told to commence shutdown process.
  2. Dispatcher halts retrieving next tasks.
  3. Teams are told to commence shutdown process.
  4. Teams log active tasks into purgatory, along with timestamps.
  5. Dispatcher shuts down and broadcasts termination.
  6. Pooled connections are shut down.
  7. New or sibling processes parse through purgatory to find tasks w/action needed.