datahub-project / datahub

The Metadata Platform for your Data Stack

Home Page:https://datahubproject.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Delta lake ingestion failed (can't parse the string value 0E-18 to decimal)

alplatonov opened this issue · comments

Describe the bug
When attempting to read or interact with a Delta table that contains a decimal datatype column with a value of 0, operations fail.
The error is "Exception: Parser error: can't parse the string value 0E-18 to decimal."
This issue prevents any interaction with tables that include such decimal values.

To Reproduce
Steps to reproduce the behaviour:

  1. Create a table in delta lake format with field decimal(38,18)
  2. try to read and see the error

Expected behaviour
The expectation is to be able to read from and interact with Delta tables regardless of the presence of decimal columns with a value of 0.

Screenshots

Additional context
pip freeze - pip.txt
data created with spark 3.4.2 and delta lake version 2.4.0