msgpack / msgpack

MessagePack is an extremely efficient object serialization library. It's like JSON, but very fast and small.

Home Page:http://msgpack.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Cannot serialize small floats / denormalized numbers?

SmooreInsight opened this issue · comments

Disclaimer: I am new to the wilderness of backend and most of the terminology I am using was learned from using Google Gemini as a tutor. If I don't make sense, that's why. I am not clear on what I'm doing, but I do want to be. Suggested learning resources for a deeper dive are welcome.

I have a float64 numpy array obtained from a bin file with the following in python:
file = bin_of_npfloat64
dt = np.dtype('float64')
arr = np.fromfile(
file,
dtype = dt
)
arr = np.ascontiguousarray(arr)
serialized_data = msgpack.packb(arr)

At first I had errors like this from trying to serialize with msgpack to later create a protobuf, but realized that my python version was 3.1 and might need updating for a lot of things.
raise TypeError("Cannot serialize %r" % (obj,))
TypeError: Cannot serialize array([9.88131292e-324, 2.76676762e-322, 7.50979782e-322,...

Now, after the update, I get this error (edited for simplicity).
File ~/anaconda3/lib/python3.11/site-packages/msgpack/init.py:36, in packb(o, **kwargs)
30 def packb(o, **kwargs):
31 """
32 Pack object o and return packed bytes
33
34 See :class:Packer for options.
35 """
---> 36 return Packer(**kwargs).pack(o)
File ... packer.pyx:294, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:300, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:297, in msgpack._cmsgpack.Packer.pack()
File ... packer.pyx:291, in msgpack._cmsgpack.Packer.pack()
...
TypeError: can not serialize 'numpy.ndarray' object'

Gemini suggests it's a limitation of msgpack being able to handle small values. The small values seemed so extremely small that I first asked my coworker if they were actually our data and not corruption or due to approximations that are unnecessary, but the more experienced coworker insisted that nothing should have changed in how the float64 is handled.

Is this a limitation of what msgpack can handle?

This is repository for msgpack spec, not for msgpack-python.

Anyway, msgpack-python doesn't support numpy.
There is msgpack-numpy. But it is not my product so I can not support it.