avrtt / datavision-demo

Real‑time/historical analytics dashboard suite unifying multi‑source enterprise data (social media, server, HR, financial, marketing, transactions) using data integration, streaming, visualization and ML techniques

Real‑time/historical analytics dashboard suite unifying multi‑source enterprise data (social media, server, HR, financial, marketing, transactions) using data integration, streaming, visualization and machine learning techniques (Apache Superset, Grafana, Kafka, Prometheus, Python, React).

This project was developed as part of my freelance work; it's basically a demo provided prior to the development of the main software. Data has been replaced with synthetic examples.

Features:

• Data integration
  Aggregates multiple data sources into a centralized system.
• Real‑time streaming
  Leverages Kafka and Prometheus for live data feeds and monitoring.
• Advanced analytics
  Utilizes machine learning algorithms for sentiment analysis and predictive insights.
• Scalable architecture
  Containerized deployment using Docker and orchestrated via Docker Compose.
• Comprehensive visualization
  Combines Apache Superset, Grafana and custom React dashboards.

The app consists of:

• Backend: a FastAPI‑based service that ingests data, processes it asynchronously and streams data via Kafka.
• Data pipeline: modules for producing and consuming Kafka messages, integrating data into a PostgreSQL database.
• Metrics: a Prometheus client exposing key performance and usage metrics.
• Visualization: integration with Apache Superset and Grafana, complemented by a React‑based custom dashboard.
• Testing & simulation: unit tests and a data simulation script that generates synthetic real‑world data.

Installation

Prerequisites:

• Docker and Docker Compose
• Python 3.9+
• Node.js and npm

Sensitive configuration values are managed via environment variables.

1. Clone:

   git clone [e-mail]:avrtt/datavision-demo.git
   cd datavision-demo

2. Backend setup:

   cd backend
   pip install -r requirements.txt
   uvicorn main:app --reload

3. Frontend setup:

   cd ../frontend
   npm install
   npm start

4. To run all services use Docker compose:

   docker-compose up --build

Dependencies

fastapi==0.95.1
uvicorn==0.21.1
SQLAlchemy==1.4.46
psycopg2-binary==2.9.6
pydantic==1.10.4
alembic==1.10.2
confluent-kafka==2.1.1
prometheus_client==0.16.0
aiofiles==23.1.0
python-dotenv==1.0.0
requests==2.28.2
textblob==0.17.1

Structure

.
├── README.md
├── .gitignore
├── docker-compose.yaml
├── requirements.txt
├── backend/
│   ├── __init__.py
│   ├── main.py
│   ├── api.py
│   ├── data_pipeline.py
│   ├── metrics.py
│   ├── db.py
│   ├── models.py
│   ├── kafka_producer.py
│   ├── kafka_consumer.py
│   ├── utils.py
│   ├── data_simulation.py
│   └── tests/
│       ├── test_api.py
│       ├── test_data_pipeline.py
│       └── test_kafka.py
└── frontend/
    ├── package.json
    ├── public/
    │   └── index.html
    └── src/
        ├── index.js
        ├── App.js
        ├── components/
        │   ├── Dashboard.js
        │   ├── MetricsChart.js
        │   └── DataTable.js
        └── services/
            ├── api.js
            └── dashboardService.js

Contribution

Contributions and issues are welcome.

License

MIT