Best way to pass on the correlation-id to child threads?
rdpravin1895 opened this issue · comments
I am running a django application using gunicorn server, and there are some child threads that will be created by the parent thread. I am getting the guid (contextvar) using get_guid method in the parent thread, passing it as an argument to the child threads, and setting the guid contextvar in the child threads using set_guid method (both of these methods are present in django_guid.api module). Only then the guid is visible in the log messages.
Is this the best way to pass on the correlation-ids to child threads? Or is there an alternate better way?
You mean your views specifically start their own threads to do work?
You mean your views specifically start their own threads to do work?
Yes. The API view itself will first return a transaction id, and delegates the work to child thread. The log messages of child thread do not contain the correlation-id (aka) guid, thats why I have to pass it using the above mentioned methods.
I see. Running these background tasks on the web server as threads is a bit unorthodox - normally Celery or similar technologies are used.
Anyway, no, contextvars are not passed between threads. You should start your thread pool with the initializer argument, and copying over contextvars. There is a good blog article about this here.
Unfortunately nothing we can build support for.
Out of curiosity @rdpravin1895, do you have a minimal code example of how it works?
Out of curiosity @rdpravin1895, do you have a minimal code example of how it works?
This is a very simple example of what we want to achieve. I have just used the Thread module here, instead of ThreadPoolExecutor. But thanks to @JonasKs, I was not aware of Celery until now, which we will start to use instead of threads.
import time
from threading import Thread
from django_guid.api import get_guid, set_guid
def long_fun(guid_id):
set_guid(guid_id)
logger.info("Entering long blocking func")
time.sleep(20)
logger.info("Sleep over")
@api_view(['POST'])
def someFunction(request):
guid = get_guid()
bg_func = Thread(target=long_fun, kwargs={'guid_id': guid})
bg_func.start()
logger.info("after starting another thread")
return JsonResponse("Response sent immediately", status=200, safe=False)
This library works very nicely with Celery, you can see how in the documentation😊 If you'd like to continue using threads, I'd recommend using the approach I linked to above.
I'm going to close this issue, but we'll still get notified if there are more comments 😊 Have a good weekend!
@JonasKs Thanks for such blazing fast responses. Your library is helping out in our large scale production observability. Kudos.
My pleasure 😊 Good luck!