Goal

University of Aveiro

The goal of this project was to implement a distributed system to process time-costly audio processing jobs, by distributing them across many remote workers.

The end-user doesn't need to wait years, but seconds.

Prominent project highlights

Fanout exchange queue - allows to send exactly one message to every subscribed client
Fault Tolerance - ACK/NACK messages, adjusting clients' heartbeats
Performance - Identifying bottlenecks is the first thing to consider to tune performance.

Dependencies

For Ubuntu (and other debian based linux), run the following commands:

sudo apt install ffmpeg
sudo apt install make

setup local python environment

Run the following commands to setup the environement:

python3 -m venv venv
source venv/bin/activate

pip install pip --upgrade
python3 -m venv --upgrade venv
pip install -r requirements.txt

setup the broker

docker-compose up -d
IMPORTANT!!! - Create an exchange queue using the broker's UI!
- Go to http://localhost:7777/#/exchanges, login with user: admin, pass: admin
- Add a new exchange
  - Name: reset
  - Type: fanout
  - Press Add exchange

Start the server

export PYTHONPATH=$PWD
make server

Start workers

You can start the workers to run on a different machine.
Before run, since they subscribe to the broker, make sure to set the broker IP in the src/config.py file. E.g. BROKER_HOST = '192.168.0.4'. By default it is localhost, if the server and the workers run on the same machine.

export PYTHONPATH=$PWD
make workers

Usage

Open the Web Page. Open templates/index.html file using browser.
Drop an MP3 file, and wait for the progress to go 100% (don't refresh the page while processing, the output will disappear), afterwards you can download your files:
- Separated drums track
- Separated vocals track
- Separated other instruments (mostly guitar + keyboard) track
- Separated bass track
- Mix of chosen instruments (default is drums + bass)

This project is licensed under the MIT License - see the LICENSE file for details.

Fault Tolerance

This system is resillient to worker crashes, please see the WORKERS_UNHEALTHY parameter to simulate the crash after WORKER_SIMULATE_CRASH_AFTER time.

Configurations

in src/config.py you can find a number of parameters to be adjusted, e.g.:

### User Conf ###
BROKER_HOST = 'localhost'           # broker IP
WORKERS_HEALTHY = 10                # Amount of workers that do not crash
WORKERS_UNHEALTHY = 2               # Amount of workers that crash (their process terminates)
WORKER_SIMULATE_CRASH_AFTER = 20    # After this amount of seconds.
### User Conf ###

Techstack

Backend: FastAPI
Middlewares: RabbitMQ
Frontend: Bootstrap, html, css

Authors

Student: Miron Oskroba

NMEC: 112169