celery-node + Python workers, prototype issue with multiple celery-node clients

[Deleted user]

Hi - first wanted to say thank you for the great work here! Great to see an actively maintained Node.js Celery option!

I'm trying to use celery-node alongside the original Python Celery project where I use a celery-node client for creating/invoking tasks but then have separate Python and Node.js workers. This will allow us to write some tasks in Python and others in Node. This means the Python tasks are registered with the Python worker, and the Node tasks are registered with the Node worker. My original hope was that it would be as easy as having a single queue for all workers with a single celery-node client, and then having the appropriate worker fulfill the task request based on which one the task was registered with. I've realized it's not quite that simple. I'm using RabbitMQ btw.

What I've tried instead is to have the celery-node worker use one queue and then the Python worker use another. I'm still trying to invoke the tasks from a single node app, so to do that I've tried instantiating two node celery clients. The issue I ran into, which may be a prototype issue, is that once I instantiate the second client with the different queue, the configuration in the first client seems to have been replaced with the config from the second. I'm not able to successfully use each client independently. check out the output below for an example. you can see the nodeClient config is overwritten with the pythonClient config as soon as the second instance is created. assuming this is not expected behavior, I can submit a formal bug ticket or provide additional details.

const celery = require('celery-node')

const nodeClient = celery.createClient('amqp://node@localhost', 'amqp://node@localhost', 'celeryNode')
console.log('node client:')
console.log(nodeClient)

const pythonClient = celery.createClient('amqp://py@localhost', 'amqp://py@localhost', 'celeryPython')

console.log('node client again:')
console.log(nodeClient)

console.log('python client:')
console.log(pythonClient)

// output:
node client:
Client {
  conf: {
    CELERY_BROKER: 'amqp://node@localhost',
    CELERY_BROKER_OPTIONS: {},
    CELERY_BACKEND: 'amqp://node@localhost',
    CELERY_BACKEND_OPTIONS: {},
    CELERY_QUEUE: 'celeryNode',
    TASK_PROTOCOL: 2
  },
  taskProtocols: { '1': [Function], '2': [Function] }
}

// *** notice the second time the node client has been overwritten with the python client config:
node client again:
Client {
  conf: {
    CELERY_BROKER: 'amqp://py@localhost',
    CELERY_BROKER_OPTIONS: {},
    CELERY_BACKEND: 'amqp://py@localhost',
    CELERY_BACKEND_OPTIONS: {},
    CELERY_QUEUE: 'celeryPython',
    TASK_PROTOCOL: 2
  },
  taskProtocols: { '1': [Function], '2': [Function] }
}

python client:
Client {
  conf: {
    CELERY_BROKER: 'amqp://py@localhost',
    CELERY_BROKER_OPTIONS: {},
    CELERY_BACKEND: 'amqp://py@localhost',
    CELERY_BACKEND_OPTIONS: {},
    CELERY_QUEUE: 'celeryPython',
    TASK_PROTOCOL: 2
  },
  taskProtocols: { '1': [Function], '2': [Function] }
}

If this is expected, is there a better way to handle routing to multiple queues from a client in a single app? Does/can node-celery support routing like Celery does ? Any advice or guidance on this general approach of having both Python and Node task workers that can be invoked from a celery-node client would be much appreciated. Thank you!

[SunMyeong Lee]

Hey!
Thank you for your interest in celery.node project!

As I can see, it's apparently a bug that is not expected to work.
I will fix it and deploy a new version today. Thank you for reporting!

I'm planning to support routing tasks in celery.node as celery does,
but I'm quite busy right now to do another personal project.
So it would be really appreciated if you make a PR for it.

Thanks.

[SunMyeong Lee]

Fixed it and published a new version.
Can you try with a new one?

[Deleted user]

@actumn thank you so much for the quick reply and fix - this resolved it, and it works perfectly now! I will certainly look into the task routing and let you know if that is something I can assist with. Thank you again!