Some examples from this Video source
For more examples and the current progress see #visualizeYouTube playlist
The logo was designed by a 3 year old girl.
YouTube pose dream is a Chrome AI extension to visualize videos. (Soon also available for Firefox.)
When you're old enough, you may remember Winamp. There is the possibility to visualize music.
However, this extension makes similar with videos. The human pose in the video is used as input for the visualization.
In the process, the existing video is transformed into a psychedelic work of art. But this depends on the video you are watching.
The goal of this project is for anyone, with or without programming experience, to be able to visualize YouTube.
In addition, animations should be interchangeable between users.
Pose estimation is done with TensorFlow.js
Particle animation is done with Proton
This project is a Google Chrome browser extension. That's why you need this browser to try it out.
- How it works
- Features
- Installation
- Usage
- Uninstall
- Build
- Further development
- License
The Chrome extension becomes active, when you watch a YouTube video.
The extension use MoveNet as model.
Visual representation of the keypoints:
Image from tensorflow.js pose-detection model
We use these keypoints to place animations or static content in the video.
After the installation you can select the following visualizations via the extension menu.
You can change the visualization while the video is playing.
Or you can change the visualization directly in the video player.
This also works in full screen mode.
With the new button the popup can be displayed or disabled.
The video Fatboy Slim ft. Bootsy Collins - Weapon Of Choice Official 4k Video was used for all screenshots.
This is TensorFlow.js in action.
MoveNet is used for the detection.
Use: all keypoints
Use: all keypoints
Use: all keypoints
Use: left_wrist, right_wrist, left_shoulder, right_shoulder, left_ankle, right_ankle, nose
Use: all keypoints
The eyes are used as a reference for displaying the image.
All use left_eye, right_eye
Every hand gets one energy ball.
Use: left_wrist, right_wrist
Two energy balls circle around the head.
Use: nose
Only the right hand gets a particle animation.
Use: right_wrist
A gravitational field is created around the nose.
Use: nose
Supernovas are constantly exploding on the nose.
Use: nose
Use: left_wrist, right_wrist
The yellow ray is connected to the right hand.
The red ray is connected to the left hand.
If the hands are too far away from the edge, the connection is interrupted.
Use: left_wrist, right_wrist, left_elbow, right_elbow, left_shoulder, right_shoulder
Use: left_wrist, right_wrist
Use: left_wrist, right_wrist
Use: left_wrist, right_wrist
Use: left_wrist, right_wrist
Use: left_wrist, right_wrist, left_elbow, right_elbow, left_shoulder, right_shoulder, left_ankle, right_ankle, left_hip, right_hip
Use: left_wrist, right_wrist, left_elbow, right_elbow, 2*left_shoulder, 2*right_shoulder, 2*left_ankle, 2*right_ankle, left_hip, right_hip, left_ear, right_ear, nose
git clone https://github.com/deep2universe/ChromePoseDream.gitOpen this URL
chrome://extensions
In the upper right corner you have to activate the developer mode.
Click this button to load the extension.
Then select the dist Folder from this repository.
You should now see the following entry.
Go to this URL
chrome://settings/accessibility
In the System settings make sure to enable this:
In the upper right corner click this to pin the extension.
Open YouTube
Watch a video and have fun.
Just go to the chrome://extensions page and disable or delete the extension.
This will be possible later via the popup/settings page.
All the code is in the src directory.
You need PARCEL for the build.
# install PARCEL
npm install -g parcel-bundler
# install dependencies
npm install
# build dist folder
npm run buildThe following is still on the TODO list:
- Support for other video platforms (e.g. Vimeo)
- Support for multi-pose detection.
- Internationalization
- Export of the pose detection keypoints for other applications.
- Use pose detection keypoints to feed GAN or VAE model and display result in webpage.
- It should be possible to deactivate the extension. Currently, you must do this in
chrome://extensions - Fill settings page with content (default animation, disable extension, language settings etc.)
- Publish to Chrome extension store.
- Clean up code.
- Add support for three.proton to enable 3D particles.
- Check content.js video event listener, - sometimes you have to reload the page to start pose dream.
- Add more particle animations.
- Create Firefox extension and publish to Firefox Browser ADD-ONS
This is an example of how this project setup could be extended in the future.
Currently, the focus of the project is on exploring the existing architectural possibilities.
With the help of artificial intelligence, many other enhancements are feasible. For example, the acoustic content of the video is not viewed and could also be used.
If you want to support, pull requests are always welcome.
If you find any bugs please report them.