Giters

regel / loudml

Loud ML is the first open-source AI solution for ICT and IoT automation

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Training Fails: Not JSON Serializable

lmangani opened this issue · comments

commented

Hello @regel & All,

Testing release 1.6.0 after a long time (installed via yum packages, managed through the grafana integration) and I noticed any successful training consistently fails being stored at the process tail with the following error, apparently regardless of the bucket settings/range:

127.0.0.1 - - [2020-10-24 15:11:25] "GET /models/hepic_hepic_statistics_calls_mean_finished__time_5m HTTP/1.1" 200 701 0.002114
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [01:00<00:00,  6.01s/it, best loss: 18.589042263891084]
ERROR:root:('Not JSON Serializable:', b'\n\x11SquaredDifference\x12\x11SquaredDifference\x1a\x17decoder_dense_2/BiasAdd\x1a\x07input_1*\x07\n\x01T\x12\x020\x01')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/loudml-1.6.0-py3.6.egg/loudml/worker.py", line 53, in run
    res = getattr(self, func_name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/loudml-1.6.0-py3.6.egg/loudml/worker.py", line 101, in train
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/loudml-1.6.0-py3.6.egg/loudml/donut.py", line 1101, in train
    model_b64 = _serialize_keras_model(self._keras_model)
  File "/usr/local/lib/python3.6/site-packages/loudml-1.6.0-py3.6.egg/loudml/donut.py", line 228, in _serialize_keras_model
    keras_model.save(path)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1211, in save
    saving.save_model(self, filepath, overwrite, include_optimizer, save_format)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/saving/save.py", line 113, in save_model
    model, filepath, overwrite, include_optimizer)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 101, in save_model_to_hdf5
    default=serialization.get_json_type).encode('utf8')
  File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/util/serialization.py", line 69, in get_json_type
    raise TypeError('Not JSON Serializable:', obj)
TypeError: ('Not JSON Serializable:', b'\n\x11SquaredDifference\x12\x11SquaredDifference\x1a\x17decoder_dense_2/BiasAdd\x1a\x07input_1*\x07\n\x01T\x12\x020\x01')
ERROR:root:job[469672de-928b-4424-b5bc-3935fdd665ad] failed: ('Not JSON Serializable:', b'\n\x11SquaredDifference\x12\x11SquaredDifference\x1a\x17decoder_dense_2/BiasAdd\x1a\x07input_1*\x07\n\x01T\x12\x020\x01')

Did anyone else experience the same and/or has a clue as of what might be causing this?

Thanks in advance!

commented

Add-on: This seems to only affect the RPM package. Using the Docker container, everything runs smoothly.