intake / intake-spark

Spark drivers for Intake

Home Page:https://intake-spark.readthedocs.io/en/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Requires pyspark >=2.3

jecolvin opened this issue · comments

While the requirements don't state a specific pyspark version is required, the use of the _to_corrected_pandas_type function from pyspark.sql.dataframe requires pyspark 2.3, as that was when that function was introduced.

Either the requirements should be updated, or just copy-paste that function into intake-spark.

Thanks for the note.
Would you be interested in supplying a PR for this change?