nteract / improved-spark-viz

🐼 WIP Improved visualizations in Spark

improved-spark-viz 🐼

WIP - This is not suitable for production use.

Improved visualizations in Spark.

Installation (Development)

Supports PySpark >= 2.2.0.

1. Clone this repo.

2. Use pip to install:

pip install -e .

Contributing

We follow the contributing guidelines of the nteract project.

References

pandas documentation

• pandas DataFrame
  • pandas.DataFrame [source]
  • pandas.DataFrame.sample [source]

PySpark API documentation

• pyspark DataFrame

  • pyspark.sql.DataFrame
  • pyspark.sql.DataFrame.sample
  • pyspark.sql.DataFrame.sampleBy

• pyspark.RDD A Resilient Distributed Dataset (RDD), the basic abstraction in Spark)

• pyspark.StorageLevel controls storage of an RDD