ClearMLCallbackHandler.on_chain_end and ClearMLCallbackHandler.on_chain_start raise exceptions of missing key

[Maciej Dzieżyc]

Describe the bug

In my langchain program, I constantly get errors with missing key output in outputs and input in inputs:

WARNING:langchain.callbacks.manager:Error in ClearMLCallbackHandler.on_chain_end callback: 'output'
WARNING:langchain.callbacks.manager:Error in ClearMLCallbackHandler.on_chain_start callback: 'input'

To reproduce

Unfortunately, I wasn't able to create a minimal example of the code which reproduces this bug.

But I did check what are the keys of incoming dictionaries before the exception occurs:

inputs.keys() in on_chain_start:

• dict_keys(['query'])
• dict_keys(['input_documents', 'question'])
• dict_keys(['question', 'context'])

outputs.keys() in on_chain_end:

• dict_keys(['text'])
• dict_keys(['output_text'])
• dict_keys(['result'])

If I'm able to narrow down the problem I'll report my findings. I hope that maybe someone had similar issues and could shed some light on the problem.

Expected behaviour

Both functions should handle different structures of dictionaries

Environment

• Server type (self hosted \ app.clear.ml): clear.ml
• ClearML SDK Version: 1.13.1
• Python Version: 3.10
• OS (Windows \ Linux \ Macos): Linux

[eajechiloae]

Hi @mdziezyc ! We are looking into this problem and #1124 and we will come back to you as soon as we have a fix.

If you could create a minimal example that would, indeed, be of great help.

[Maciej Dzieżyc]

The only thing I can add for now is that I checked W&B and it worked well, so it does not seem to be something specific related to my code. If I find sth more, I'll let you know.

[eajechiloae]

@mdziezyc we have submitted a PR related to this issue and #1124 : langchain-ai/langchain#11472

[pollfly]

Hey @mdziezyc! Just letting you know that this issue has been resolved. See langchain-ai/langchain#11472. Let us know if there are any issues :)