task.connect() behavior when running on remote

[Liron Ilouz]

Describe the bug

When a task is running remotely, a connected object cannot be updated

To reproduce

1. Use the example from https://github.com/allegroai/clearml/blob/master/examples/reporting/hyper_parameters.py.
2. Modify it to execute remotely as following

# ClearML - example code for logging into "CONFIGURATION":
# - ArgumentParser parameter logging
# - user properties logging
# - logging of hyperparameters via dictionary
# - logging of hyperparameters via TaskParameters
# - logging of configuration objects via TaskParameters
#
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import sys
from argparse import ArgumentParser
from enum import Enum


from clearml import Task
from clearml.task_parameters import TaskParameters, param, percent_param


# Connecting ClearML with the current process,
# from here on everything is logged automatically
Task.add_requirements('clearml')
task = Task.init(project_name='FirstTrial', task_name='first_trial')
task.execute_remotely(queue_name='default')


# -----------------------------------------------
#  Report user properties
# -----------------------------------------------

task.set_user_properties(custom1='great', custom2=True)
task.set_user_properties(custom3=1, custom4=2.0)


# -----------------------------------------------
#  Report hyperparameters via dictionary
# -----------------------------------------------

class StringEnumClass(Enum):
    A = 'a'
    B = 'b'


class IntEnumClass(Enum):
    C = 1
    D = 2

parameters = {
    'list': [1, 2, 3],
    'dict': {'a': 1, 'b': 2},
    'tuple': (1, 2, 3),
    'int': 3,
    'float': 2.2,
    'string': 'my string',
    'IntEnumParam': StringEnumClass.A,
    'StringEnumParam': IntEnumClass.C
}
parameters = task.connect(parameters)

# adding new parameter after connect (will be logged as well)
parameters['new_param'] = 'this is new'

# changing the value of a parameter (new value will be stored instead of previous one)
parameters['float'] = '9.9'
print(parameters)

Expected behaviour

The value of parameters['float'] should change to '9.9' bit it actually stays 2.2

Environment

• Server type - app.clear.ml
• ClearML SDK Version - clearml==1.14.4
• Python Version - 3.9.16
• Dockerized worker

Related Discussion

https://clearml.slack.com/archives/CTK20V944/p1709474983437769