Requires pyspark >=2.3
jecolvin opened this issue · comments
While the requirements don't state a specific pyspark version is required, the use of the _to_corrected_pandas_type
function from pyspark.sql.dataframe
requires pyspark 2.3, as that was when that function was introduced.
Either the requirements should be updated, or just copy-paste that function into intake-spark.
Thanks for the note.
Would you be interested in supplying a PR for this change?