model write: pickled pandas objects aren't backward competible, when updating pandas
alebaran opened this issue · comments
Not a big thing, but I have noticed that when I've upgraded pandas from 0.25.3 to 1.0.4, I wasn't able to load previously saved model versions. Object structure probably changed a bit and it wasn't loading correctly. I was able to reproduce the issue by using
pickle.dump( pd.DataFrame(['a','b'],index=pd.MultiIndex.from_tuples(((1,2,3),(2,4,5)))), open( "try", "wb" ) )
with the older version and
pickle.load(open(r'try', "rb"))
with the newer one.
Pandas itself provides pickling function, which seems to ensure backward compatibility. The following works:
pd.DataFrame(['a','b'],index=pd.MultiIndex.from_tuples(((1,2,3),(2,4,5)))).to_pickle('try')
with the older version and
pd.read_pickle('try')
with the newer one.
It would be nice to have to use native pandas pickling.
read_pickle seems to have exception handling for backward compatibility
https://github.com/pandas-dev/pandas/blob/v1.0.4/pandas/io/pickle.py#L187
This can't be fixed as DataFrames are not pickled individually in modelx
I guess they also rarely change the structure
This was rather a critical issue so I decided to fix it.