leogau / rusty-celery

Rust implementation of Celery for producing and consuming background tasks

Home Page:https://rusty-celery.github.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool


A Rust implementation of Celery for producing and consuming asyncronous tasks with a distributed message queue.


Build License Crates Docs Help wanted Discord


We welcome contributions from everyone regardless of your experience level with Rust. For complete beginners, see HACKING_QUICKSTART.md.

If you already know the basics of Rust, the Rusty Celery Book is the best place to start. If you're coming from Python you may also be interested to know what's different. And if you've never heard of Celery, the official Celery Project is a great source of tutorials and overviews.

Quick start

Define tasks by decorating functions with the task attribute.

use celery::TaskResult;

#[celery::task]
fn add(x: i32, y: i32) -> TaskResult<i32> {
    Ok(x + y)
}

Create an app with the app macro and register your tasks with it:

let my_app = celery::app!(
    broker = AMQP { std::env::var("AMQP_ADDR").unwrap() },
    tasks = [add],
    task_routes = [
        "*" => "celery",
    ],
);

Then send tasks to a queue with

my_app.send_task(add::new(1, 2)).await?;

And consume tasks as a worker from a queue with

my_app.consume().await?;

The ./examples directory contains a simple Celery app that is implemented in both Rust (celery_app.rs) and Python (celery_app.py) using an AMQP broker.

If you already have an AMQP broker running you can set the environment variable AMQP_ADDR to your broker's URL (e.g., amqp://localhost:5672//, where the second slash at the end is the name of the default vhost). Otherwise simply run the helper script:

./scripts/brokers/amqp.sh

This will download and run the official RabbitMQ image (RabbitMQ is a popular AMQP broker). Then you can run a Rust worker with:

cargo run --example celery_app consume

And then send tasks to the worker with:

cargo run --example celery_app produce

Alternatively, you can consume or produce tasks from Python by running

python examples/celery_app.py consume

Or

python examples/celery_app.py produce

You'll need to have Python 3 installed, along with the requirements listed in the requirements.txt file.

Road map and current state

βœ… = Supported and mostly stable, although there may be a few incomplete features.
⚠️ = Partially implemented and under active development.
πŸ”΄ = Not supported yet but on-deck to be implemented soon.

Core

Status Tracking
Protocol ⚠️
Producers βœ…
Consumers βœ…
Brokers βœ…
Backends πŸ”΄
Beat πŸ”΄
Baskets πŸ”΄

Brokers

Status Tracking
AMQP βœ…
Redis πŸ”΄

Backends

Status Tracking
RPC πŸ”΄
Redis πŸ”΄

Team

Rusty Celery is an open source community effort. It is also backed by Structurely, a start-up building conversational AI that has been using Python Celery in production since 2017.

About

Rust implementation of Celery for producing and consuming background tasks

https://rusty-celery.github.io

License:Apache License 2.0


Languages

Language:Rust 98.9%Language:Makefile 1.0%Language:Shell 0.1%