PrachpaveenY / PySpark-project

PySpark-project

PySpark-project (Complete)

• Project : PySpark Project
• Tools = Python, PySpark, Pandas, Azure SQL Database

Detail :

• Create File : CSV, TXT.
• Use PySpark and Pandas : manage data.
• microsoft azure project : use manage data of Azure SQL Database

Structure :

📁 PySpark-project
└── 📁 all project
|   └── 📁 apache airflow
|   |
|   └── 📁 app
|   |   └── 📁 tables                           // All CSV Files
|   |   └── 📄 use.ipynb                        // Main File of app ( use manage data )
|   |
|   └── 📁 google cloud platform
|   |
|   └── 📁 microsoft azure project
|   |   └── 📄 Azure-sql-database-project.ipynb // Main File of microsoft azure project ( use manage data of Azure SQL Database )
|   |
└── 📁 img
└── 📁 tutorial
|   └── 📄 HR01.csv                         // Use CSV of tutorial
|   └── 📄 Pandas_Tutorial1.ipynb           // Pandas Tutorial
|   └── 📄 PySpark_Tutorial1.ipynb          // PySpark Tutorial
|   └── 📄 Python_Tutorial01.ipynb          // Python Tutorial
|   └── 📄 python_tutorial01.py             // Python Tutorial
|
└── 📄 HR01.csv
└── 📄 hr01.txt
└── 📄 PySpark_createData.ipynb           // Create File
└── 📄 README.md

Run project step :

• Run use.ipynb
• Run PySpark_Tutorial1.ipynb