cross-dill-version function serialization problems
benclifford opened this issue · comments
I've encountered a few issues along with the Globus Compute developers where functions pickled with one version of dill will not unpickle with another version (0.3.5.1 vs 0.3.6) -- I'm unclear whether this is an intended incompatibility or a bug.
If it's intended incompatibility, we'll use that to inform our future packaging and user requirements; if it's a bug, I'll put some more effort into debugging it and update this issue.
Serialized objects should be compatible across different versions of dill. This is the case, and has been generally the case across all versions. There have been a few exceptions where certain objects break across version change. This has been generally due to a change from a PR that removes (or otherwise modifies) a dill internal helper function or variable which is used to (un)pickle an object.
As it started getting harder and harder to preserve all dill internals across bug fixes and feature improvements, the dill._shims mechanism that provides a way to "shim" in legacy-version dependent behavior. So, in theory, there shouldn't be any breakage across current and future dill versions, and if breakage is found, one can add a shim.
Feel free to reopen this if needed.