rvm-xx / sparklingpandas

Pandas On PySpark(POPS)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

SparklingPandas

SparklingPandas aims to make it easy to use the distributed computing power of PySpark to scale your data anlysis with Pandas.

Documentation

None (right now). You can find some slides in the slides/ directory

Requirements

The primary requirement of SparklingPandas is that you have a recent (v1.0-SNAPSHOT currently) version of Spark installed - http://spark.apache.org

Using

Make sure you have the SPARK_HOME enviroment variable set correctly, as SparklingPandas uses this for including the PySpark libraries

Other than that you can install SparklingPandas with pip and just import it. The primary unit of SparklingPandas is a PRDD (Pandas Resillent Distributed Data Set)

State

This is in early development and should not be considered usable.

Support

Check out our Google group at https://groups.google.com/forum/#!forum/sparklingpandas

About

Pandas On PySpark(POPS)

License:Apache License 2.0


Languages

Language:Python 98.8%Language:Shell 1.2%