Developed in Central India's largest Hackathon - WittyHacks!
NOTE: This project is just one part of what we did at the hackathon. We also went on to achieve a decent success on cracking the data for the Problem number 4 of the contest. Due to the organisation's privacy policy, we won't be updating the code for Problem 4 (which was data analysis and machine learning applied on historical data collected on wiityfeed).
A collection of lot useful utilities ranging from editing the background of videos using recurrent neural networks to landmark detection for masking and other purposes.
- Description
- Salient Features
- Technologies used
- How to use Videofy?
- How it works?
- How to build from Source?
- Contributors
- Future Work
Our orginal aim was to take video input and process them to edit the background and other features in the video, but then we ended up doing a lot better on some fronts while we had to compromise on some!
We weren't really sure about whether we can do that in real time with the little computation, we have. And, then, there was a time, when the computation time was very high and we spend a lot of time on to rethink our approach. The hackathon more of became like a research and we discovered a lot of failed approaches in the way. We will discuss more about the challenges later on.
But, before I proceed, let me mention of an important use case of this project. Like, people can create inner voice videos at their home without even thinking more about the background and other related things.
- Takes a video and then processes it to generate another video with the desired background. The users can choose their desired backgrounds (at this moment, this can be done by tweaking the code).
- We use mask recurrent-convulational neural network for the detection of a subject and differentiating it with the background.
- Realtime videos can be generated (with a small lag) - given the internet connectivity is good.
- Higher quality videos can be generated by submitting the video to our portal, where all the computations will happen. The process will take time - because better the quality - more the time and accordingly the generated videos can be then used further!
- The user can also generate the sketched videos of himself. This is comparitively quite faster when compared to the colored quality videos.
- Talking Cartoon - a feauture aligned quite closely with how talking tomcat works. So, yes, we have also made an android app where you can record the audio and then as an output the video will be there with a cartoonized character of the man with the same voice.
It is as simple as searching on google and obviously a lot more easier than coming to the Inner Voice studio. You can simply open our portal and upload your video there and then soon after processing the modified file will be ready for the download.
Really simple, isn't it? :)
Videofy uses a number of open source projects to work properly:
- [opencv-python] - Recurrent convulational neural networks
- [nltk-python] - For sentiment analysis
- [coco] - large image dataset for training
- [numpy-python] - For math function
- [socket] - For client-server communication
- [materialize.css, html5] - For designing web page
You might be curious to know how this tool works. There do not exist many tools with such a use case and offering such an ease to the users.
We use various specialized techniques in order to achieve our objective. And, here are the details of the same :
- The first task is to get the frames from the video and then compute on the frames on one-by-one basis to transform them with a different background.
- For every frame, we use R-CNN to mask our subject.
- Then, we simply craft a video out of it by gluing the frames together.
- Tadaaa, the video will be ready for consumption.
- We already have the website live on our local server.
- An android app can be made for this tool easily.