_create_code in dill 0.3.5.1 cannot deserialize create code invocations from dill 0.3.6, 0.3.7 in Python 3.10

[Ben Clifford]

If I serialize a function, giving a pickled _create_code call in the resulting pickle, I experience this cross version incompatibility:

create such a pickle with dill 0.3.6 and 0.3.7, on Python 3.10
unpickle with dill 0.3.5.1, on Python 3.10

observe this stack trace:

Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/dillrun.py", line 6, in <module>
    func = dill.load(file=file)
  File "/home/benc/.local/lib/python3.10/site-packages/dill/_dill.py", line 373, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/home/benc/.local/lib/python3.10/site-packages/dill/_dill.py", line 646, in load
    obj = StockUnpickler.load(self)
  File "/home/benc/.local/lib/python3.10/site-packages/dill/_dill.py", line 805, in _create_code
    return CodeType(args[0], 0, 0, *args[1:])
TypeError: code expected at most 16 arguments, got 19

Small reproduction:

Run this script under python 3.10 with dill 0.3.6 or dill 0.3.7:

$ cat dillgen.py 

import dill

def myfunc():
    return 7

with open("f-"+dill.__version__+".dill", "wb") as file:
    dill.dump(myfunc, file=file)

then change to dill 0.3.5.1 and run this script:

$ cat ../parsl/dillrun.py 
import sys
import dill


with open(sys.argv[1], "rb") as file:
    func = dill.load(file=file)

assert func() == 7

$ python3 dillrun.py f-0.3.6.dill 
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/dillrun.py", line 6, in <module>
    func = dill.load(file=file)
  File "/home/benc/.local/lib/python3.10/site-packages/dill/_dill.py", line 373, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/home/benc/.local/lib/python3.10/site-packages/dill/_dill.py", line 646, in load
    obj = StockUnpickler.load(self)
  File "/home/benc/.local/lib/python3.10/site-packages/dill/_dill.py", line 805, in _create_code
    return CodeType(args[0], 0, 0, *args[1:])
TypeError: code expected at most 16 arguments, got 19

(this follows on from this question: #608 which I think suggests this is unexpected behaviour)

[Ben Clifford]

The workaround Globus Compute is now using for this (vs the need for dill >=0.3.6 for recent python versions) is to pin specific dill versions based on Python version: 0.3.5.1 for python 3.10 and earlier, and 0.3.6 for python 3.11 and later.

https://github.com/funcx-faas/funcX/pull/1259/files

[Mike McKerns]

python had backward incompatible changes for code objects across those versions for 3.10 and above. I was able to add workarounds for deserializing pickles from older versions -- there's nothing that can be done to support deserializing pickles from newer versions with older versions.

I'm closing this as a won't fix.