Support ingesting small parquet files
cmenguy opened this issue · comments
Currently the small file API support in ingestion seems incomplete, as in the method uploadSmallFile it expects data of type Union[list, dict] which works fine for JSON, but for Parquet format you would be passing in bytes, I couldn't figure out how to get this working with some Parquet data so ended up using JSON. But I think either the prototype of the function needs to change since the notion of multiline isn't really applicable to Parquet, or we need to actually change the code to fully support Parquet binary payload.
