Question - Communicate metadata with message

[Dimitri Saingre]

Hi,
I'm looking to implement a simple OpenTelemetry (OT) instrumentation to have distributed tracing between my actors (the goal is to find out where are the performances issues I have on another project using Thespian). I have several actors communicating with each other and basically, I want to trace the different messages exchange (ex: actor A received a message that took 2ms to process and then sent a message to actor B that took 1ms to process...).

The way OT works, I would need to propagate a context between the different actors, along with the messages exchanged (so OT can group the different calls together). I wanted to know if there were any way to send this context as a metadata along with the message I send between two actors.

A quick and dirty way to implement this would be to bundle the message with the context. The issue I have with this is that it impact all the application (as every actor now needs to separate the tracing context and the received message). That means that I need two implementations of my app if i want the tracing to be optional. That's why I thought that sending the context as a metadata would be "cleaner".

Does the current implementation of Thespian have anything like his? If not, would you consider it worth implementing for Thespian (I could try to implement it).

Thanks

[Kevin Quick]

I had not been aware of the OpenTelemetry project previously. Based on a quick review, it looks potentially interesting, but I would want to review it in a bit more time and depth before committing to providing Thespian core support for it. If you have OT expertise and a desire to move in this direction I'd be happy to further explore this with you (kq1xquixck s/x// via gmail).

At the present time, Thespian has an internal thesplog that is used for internal debugging, and support for standard python logging. One option would be to use logging information to extract the tracing since Thespian will forward all logging information to a central log (a very rough analog to an OT Collector).

Another way to approach this would be to create a wrapper decorator... something along the lines of:

def withTelemetry(f):
  orig_handler = f.receiveMessage
  orig_send = f.send
  def telemetryHandler(self, message, sender):
    if hasattr(message, '.telemetry'):
      do_something_with_incoming(getattr(message, '.telemetry'))
      delattr(message, '.telemetry')
    return orig_handler(actor, message, sender)
  def telemetrySender(self, message, target):
    setattr(message, '.telemetry', generate_telemetry_info())
    orig_send(self, message, target)
  f.receiveMessage = telemetryHandler
  f.send = telemetrySender
  return f

[note: the above is a quick, untested sketch of a decorator that should be sufficient to demonstrate the general functionality, but which may not be 100% correct on the details. Please see thespian/transient.py and thespian/troupe.py for active examples of decorators of this style.]

Then your actors could be written without intrusive impact:

@withTelemetry
def myActor(Actor):
  ...

The wrapper can then enhance messages with any useful information for telemetry, and extract telemetry information from incoming messages. [Pending further familiarity with OT, I suspect this might be my approach to supporting OT as well.]