leomaurodesenv / pyspark-study

PySpark study - some challenges and their solutions

PySpark Study

This reposotory contains a simple coding to study Spark.

• PySpark is the Python API for Spark.

---

Challenges

1. Read a CSV file and convert columns to rows; simulating a columnar table.
2. Removing duplicates based on id and last update update_date.
3. Mapping the data type based on a JSON configuration file.

---

Coding

Suprise yourself viewing project.py.

• Install PySpark: Spark, PySpark, and so on.

#-- run
$ python project.py

---

Also look ~

• License MIT
• Create by Leonardo Mauro ~ leomaurodesenv