usc-isi-i2 / rltk

Record Linkage ToolKit (Find and link entities)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Record class needs to be defined in a module in distributed mode

GreatYYX opened this issue · comments

This issue is caused by incomplete serialization of Record class defined in __main__.

cloudpipe/cloudpickle#243

Two possible solutions here:

  • let upstream provide the feature to pickle interactively defined property type and the instance of the type.
  • use another way to indicate if the property should be cached: using a function decorator to update indicators in __dict__ instead of using self-defined type of property.

Upstream provided another solution: override __reduce__ in cached_property:

def __reduce__(self):
    return cached_property.__new__, (cached_property, ), {'func': self.func}