praveendecode / iss-data-warehouse-mongodb-sql-project

Space Exploration Data Fusion : Unleashing the International Space Station Insights with MongoDB and SQL Integration

International Space Station Data Harvesting and Warehousing (Live Data)

Overview

• In this project, the raw API data of the International Space Station (ISS) is transformed into MongoDB documents and SQL records. The project aims to unlock valuable insights that are helpful for communication timestamp analysis, message content analysis, and more. This is achieved through the effective storage capabilities of MongoDB's document-oriented storage and SQL's relational database management.

Features

• Data Transformation: Raw ISS API data is transformed into MongoDB documents and SQL records.

• Storage Efficiency: MongoDB's document-oriented storage and SQL's relational database management provide efficient data storage solutions.

Getting Started

Step 1: Install/Import Required Libraries

• Ensure the necessary libraries are installed or imported: Streamlit, Pandas, JSON, PyMongo, Requests, Time, and Psycopg2.

Step 2: Utilize the 'API2MongoSQL' Class

• Inside the class, there are six methods that handle the transformation and management of data:

• API iss data converted into MongoDB Documents

• Fetch MongoDB Documents

• Converting MongoDB Documents Into SQL Records

• Fetch SQL Table Records

• Delete Document in Mongo Collection

• Delete SQL Records

• Note : Streamlit is integrated for GUI purposes.

Step 3: Run the Project with Streamlit

• Open the command prompt in the directory where the "DataSyncPro.py" file is located. Execute the command: streamlit run DataSyncPro.py. It will open in a web browser.4

Step 4: Configure Databases

• Make sure to connect your MongoDB Atlas and PostgresSQL DBMS in your local setup.

Technical Skills

• Python (Scripting)

• Data Collection

• MongoDB

• SQL

• API Integration

• Data Management using MongoDB (Atlas) and PostgresSQL

• Integrated Development Environment (IDE): PyCharm Community Version