M. Shafiq Safian's repositories
0103_Streaming_Scheduler_Template
Creating a template for airflow code block & structure to expedite process of building scheduler
analyst-ingestor-postgres
A tools for analyst to ingest data dedicated for vizualization , they dont have any access to other schema which avoid the danger for deleting other important data) . They dont have to access to dbeaver (displaying overall schema ) which involve risk to other schema for other analyst
real-estate-end2end-pipes
Run end to end pipeline to scrape lelong & real estate deals according to states and postcode
spark-ETL-component-library
The goal of CLAIMED is to enable low-code/no-code rapid prototyping style programming to seamlessly CI/CD into production.
streamlit-converter-deploy-instances
Deploy live streamlit for analyst to utilize specific to maintain leading 000 and control read how many rows.
xlsx_to_json
Simple apps to convert xlsx files to json with option nrows for speeds , dtypes as string to avoid missing leading 000 when conversion
airflow_standard
Apache Airflow tutorial
BobbyAxelrods
Config files for my GitHub profile.
Bridge-Forward2
Forward message automatically from channel/group with telegram bot @/heroku
BridgeDaoJobAutomate
This is Telegram Messages Forwarder Userbot by @AbirHasan2005
data-engineering-zoomcamp
Free Data Engineering course!
databricks-labs
The resources of the preparation course for Databricks Data Engineer Associate certification exam
dataengineering_template
Template to setup infra for any typical data engineering project to reduce hassle in running adhoc task using open source tool intergrated with cloud service. (AWS) and soon will be available for azure too.
dosm_cleaning_etl
custom etl cleaning
dosm_consistency_etl
etl for consistency
etl_01
Testing full end to end etl pipeline complete with log format for other future submission
etl_spark_airflow_emr
Capstone project of the data engineer course at Udacity
HDFS_Project
The goal is to develop an intuitive platform where users can search for Airbnb apartments based on a target city, budget, and duration of stay, all powered by the intelligent language model, GPT-3.
HDFS_SETUP_2
This project aims to move the data from a Relational database system (RDBMS) to a Hadoop file system (HDFS)
market-analytics
analysis to finalize decision to proceed with stocks or crypto market
Opensource-data-engineering
An Awesome List of Open-Source Data Engineering Projects
pandas_module
Practice your pandas skills!
Robby-chatbot-features
Added new feature in this chatbot (chat through chatbot by uploading scanned pdf)