Don't forget to hit the ⭐ if you like this repo.
The information on this Github is part of the materials for the subject High Performance Data Processing (SECP3133). This folder contains general Exploratory Data Analysis (EDA) information as well as EDA case studies using Malaysian datasets. This case study was created by a Bachelor of Computer Science (Data Engineering), Universiti Teknologi Malaysia student. In addition, my research group also contributed materials and case studies. Thank you to the collaborators who shared their knowledge in this github.
- Python for beginners
- Web scraping and Python web framework
- Exploratory data analysis
- Big data processing
Exploratory Data Analysis (EDA) is a crucial step in the data science process. It involves the visual and statistical examination of a dataset to better understand its characteristics, patterns, and relationships.
✅️ EDA involves using graphics and visualizations to explore and analyze a data set. The goal is to explore, investigate and learn, as opposed to confirming statistical hypotheses.
✅️ EDA is used by data scientists to analyze and explore datasets and summarize the main characteristics of them.
✅️ EDA makes it easier for data scientists to discover patterns, spot anomalies, test a hypothesis, or check assumptions.
✅️ EDA is primarily used to provide a better understanding of dataset's variables and their relationships.
✅️ EDA can also help determine whether the statistical techniques you are considering are appropriate for data analysis.
✅️ Developed by the American mathematician John Tukey in the 1970s, EDA techniques continue to be a widely used method in the data exploration process today.
✅️ The main purpose of EDA is to help you look at the data before making any assumptions. In addition to better understanding the patterns in the data or detecting unusual events, it also helps you find interesting relationships between variables.
✅️ Data scientists can use exploratory analysis to ensure that the results they produce are valid and relevant to desired business outcomes and goals.
✅️ EDA also helps stakeholders by verifying that they are asking the right questions.
✅️ EDA can help to answer questions about standard deviations, categorical variables, and confidence intervals.
✅️ After the exploratory analysis is completed and the predictions are determined, its features can be used for more complex data analysis or modeling, including machine learning.
👉 Python is a popular programming language for data science and has several libraries and tools that are commonly used for EDA such as:
- Pandas: a library for data manipulation and analysis.
- Numpy: a library for numerical computing in Python.
- Matplotlib: a plotting library for creating visualizations.
- Seaborn: a library based on matplotlib for creating visualizations with a higher-level interface.
- Plotly: an interactive data visualization library.
In EDA, you might perform tasks such as cleaning the data, handling missing values, transforming variables, generating summary statistics, creating visualizations (e.g. histograms, scatter plots, box plots), and identifying outliers. All of these tasks can be done using the above libraries in Python.
- developers.google: Good Data Analysis
- Datascience using Python: Exploratory_Data_Analysis
- Towardsdatascience: What is Exploratory Data Analysis?
- Wikipedia: Exploratory data analysis
- r4ds: Exploratory Data Analysis
- careerfoundry:What Is Exploratory Data Analysis?
- How To Conduct Exploratory Data Analysis in 6 Steps
- A Five-Step Guide for Conducting Exploratory Data Analysis
- I asked ChatGPT to do Exploratory Data Analysis with Visualizations
- simplilearn: What is Exploratory Data Analysis? Steps and Market Analysis
- Exploratory Data Analysis (EDA): Types, Tools, Process
- projectpro: Exploratory Data Analysis in Python-Stop, Drop and Explore
- medium.com: 10 Things to do when conducting your Exploratory Data Analysis (EDA)
- towardsdatascience.com: An Extensive Step by Step Guide to Exploratory Data Analysis
- EDA - Exploratory Data Analysis: Using Python Functions
- Step-by-Step Exploratory Data Analysis (EDA) using Python
- Exploratory Data Analysis Tutorial | What Is EDA | How EDA Works | EDA In Python | Intellipaat
- Live Day 1-Live Session On EDA And Feature Engineering- Zomato Dataset
- Live Day 2-Live Session On EDA And Feature Engineering- Black Friday Dataset
- Live Day 3-Live Session On EDA And Feature Engineering- Flight Price Prediction Dataset
- Step By Step Process In EDA And Feature Engineering In Data Science Projects
- Exploratory Data Analysis(EDA) of Titanic dataset
- Exploratory Data Analysis (EDA) Using Python | Python Data Analysis | Python Training | Edureka
- Exploratory Data Analysis with Pandas Python
- How to Do Data Exploration (step-by-step tutorial on real-life dataset)
- Exploratory Data Analysis (Step by Step)
- A Simple Tutorial on Exploratory Data Analysis
- Intro to Exploratory data analysis (EDA) in Python
- Topic 1. Exploratory Data Analysis with Pandas
- Detailed exploratory data analysis with python
- EDA using Python Pandas
- Pandas: EDA of Cars Dataset
- Step-by-step Data Preprocessing & EDA
- PacktPublishing/Hands on Exploratory Data analysis with Python
- code4kunal/eda-python-examples
- SouRitra01/Exploratory-Data-Analysis-EDA-in-Banking-Using-Python
- sandipanpaul21/EDA-in-Python
- vharivinay/python-eda-viz
- demonpratapdemon/Exploratory-Data-Analysis-EDA-and-PreProcessing
- PacktPublishing/Python-for-Data-Analysis-step-by-step-with-projects-
- sandyy2505/Cardio Good Fitness Project
- ajaymache/Data analysis of used car database
| No | Dataset | Colab | GitHub |
|---|---|---|---|
| 1 | Boston |  |
 |
| 2 | Car Features and MSRP | |
|
| 3 | Housing Dataset | |
|
| 4 | United Nations Development Corporation | |
|
| Team | Title | Colab | GitHub |
|---|---|---|---|
| 404 Error | Property in Kuala Lumpur | |
|
| Alrite | The Exportation of Plantation in Sarawak | |
|
| BEFE | Covid-19 Clusters in Malaysia | |
|
| Boboiboy | Property Listings in Kuala Lumpur | |
|
| COLBY | Malaysia GE-14 Result | |
|
| FANTOM | Daily recorded COVID-19 cases at state level In Malaysia | |
|
| HAHA | Foreign Direct Investment In Malaysia | |
|
| HD | Guna Tanah Tampin 2021 | |
|
| KIA | Malaysia State Election 2018 | |
|
| LAB | Malaysia Air Pollution Analysis | |
|
| MAAM | Malaysia Hospital Patient Movement Analysis | |
|
| MEOW | Capacity and utilisation of Intensive Care Unit (ICU) beds during COVID-19 | |
|
| MM | Malaysia's 14th State Election Result | |
|
| PIXALATED | Number of deaths in Malaysia from 2001 to 2018 | |
|
| POTATO | Death by state, sex and age group Malaysia 2001-2018 | |
|
| QnX | Real Estate Kuala Lumpur Malaysia | |
|
| SAMVERSE | Restaurant Rating in Malaysia | |
|
| SMOL | Population in Malaysia from 2010-2019 | |
|
| SQ | Number of Cases and Incidents Rate of Communicable Disease by State | |
|
| TUK | Number of Government School Pupils by District Education Office and State 2017-2018 | |
|
| UWU | Property Listings in Kuala Lumpur | |
|
| Name | Title | Colab | GitHub |
|---|---|---|---|
| Li Jing | ABC | |
|
| Saleh Dhekre Saber Saleh | ABC | |
|
| Eman Al Jabarti | ABC | |
|
| Anwar Said Salim Al Talaii | ABC | |
|
| Zhu Caihua | ABC | |
|
| Shiekhah AL Binali | ABC | |
|
| Li Haopeng | ABC | |
|
EDA is a vital but time-consuming task in a data project. Here are 10 open-source tools that generate an EDA report in seconds.
| Library | Description | Web | Github |
|---|---|---|---|
| SweetViz | - In-depth EDA report in two lines of code. - Covers information about missing values, data statistics, etc. - Creates a variety of data visualizations. - Integrates with Jupyter Notebook. |
🌐 |  |
| Pandas-Profiling | - Generate a high-level EDA report of your data in no time. - Covers info about missing values, data statistics, correlation etc. - Produces data alerts. - Plots data feature interactions. |
🌐 | |
| DataPrep | - Supports Pandas and Dask DataFrames. - Interactive Visualizations. - 10x Faster than Pandas based tools. - Covers info about missing values, data statistics, correlation etc. - Plots data feature interactions. |
🌐 | |
| AutoViz | - Supports CSV, TXT, and JSON. - Interactive Bokeh charts. - Covers info about missing values, data statistics, correlation etc. - Presents data cleaning suggestions. |
🌐 | |
| D-Tale | - Runs common Pandas operation with no-code. - Exports code of analysis. - Covers info about missing values, data statistics, correlation etc. - Highlights duplicates, outliers, etc. - Integrates with Jupyter Notebook. |
🌐 | |
| dabl | - Primarily provides visualizations. - Covers wide range of plots: Scatter pair plots. Histograms. - Target distribution. |
🌐 | |
| QuickDA | - Get overview report of dataset. - Covers info about missing values, data statistics, correlation etc. - Produces data alerts. - Plots data feature interactions. |
🌐 | |
| Datatile | - Extends Pandas describe(). - Provides column stats: column type count, missing, column datatype. - Mostly statistical information. |
🌐 | |
| Lux | - Provides visualization recommendations. - Supports EDA on a subset of columns. - Integrates with Jupyter Notebook. - Exports code of analysis. |
🌐 | |
| ExploriPy | - Performs statistical testing. - Column type-wise distribution: Continuous, Categorical - Covers info about missing values, data statistics, correlation etc. |
🌐 | |
Please create an Issue for any improvements, suggestions or errors in the content.
You can also contact me using Linkedin for any other queries or feedback.