A scalable YouTube video processing pipeline that seamlessly downloads YouTube videos, processes them with FFmpeg, and uploads the refined videos to Cloudinary ( or S3 etc ..). This project uses Celery for distributed task processing and RabbitMQ as a message broker, ensuring an efficient and scalable workflow.
- YouTube video downloading using Pytube
- Video processing with FFmpeg for various resolutions
- Uploading processed videos to Cloudinary with detailed logging
- Scalable architecture with Celery and RabbitMQ
- Containerized application setup with Docker and docker-compose
- MongoDB integration for data storage and change streams monitoring
Before you start, ensure to configure necessary environment variables in the `.env` file. Use the `.env.example` as a template to create your `.env` file with the appropriate values.
Navigate to the project's root directory and execute the following command:
docker-compose up --build
Scale the project and increase the processing by initiating more Celery worker instances using the `--scale` option:
docker-compose up --scale celery=3
Once the application is running, you can send POST requests to the endpoint http://localhost:5000/process_video to initiate the video processing task.
You can use the following curl command as an example to send a request:
curl -X POST -H "Content-Type: application/json" -d '{"video_url": "https://www.youtube.com/watch?v=RQBqBTUY9Hw","preferred_resolution":"720p"}' http://localhost:5000/process_video
Manage and monitor the message queues by accessing the RabbitMQ management interface at `http://localhost:15672\` ( Default user and password are "guest").
Feel free to fork the project, create a feature branch, and send me a pull request.
This project is licensed under the 0BSD License.
For more information, feel free to contact me.