harupy / pyspark-util

PySpark Utils

pyspark-util

A set of pyspark utility functions.

Example:

import pyspark_util as psu

data = [(1, 2, 3)]
columns = ['a', 'b', 'c']
df = spark.createDataFrame(data, columns)
prefixed = psu.prefix_columns(df, 'x')
prefixed.show()

Output:

+---+---+---+
|x_a|x_b|x_c|
+---+---+---+
|  1|  2|  3|
+---+---+---+

Installation

pip install pyspark-util

Development

Setup

docker-compose build
docker-compose up -d

Lint

docker exec psu-cnt ./tools/lint.sh

Test

docker exec psu-cnt ./tools/test.sh