Harmonize Project for Philips Hue

Harmonize Project is a low latency video analysis and pass-through application built in Python which alters Philips Hue lights based on their location relative to a screen; creating an ambient lighting effect and expanding content past the boundaries of a screen.

Check out our Reddit thread here and watch the demo below! Electromaker explains how our application works at a high level in his podcast here!

GitHub deleted our account at 250 stars! Prior to deletion, we were in the top 20,000 repositories and top 7,000 Python-based projects on GitHub. Sorry for the interruption and loss of issues.

Harmonize Project (formerly known as Harmonize Hue) has no affiliation with Signify or Philips Hue. Hue and Philips Hue are trademarks of Signify.

Features:

Light color and intensity changes based on pixels in its relative set location
Video -> Light latency of 80ms via Streaming to Hue Lights via Entertainment API
Sending 50-75 color updates per second

Requirements

Hardware: (Tested on Raspberry Pi 4B)

RAM: 256MB Free Minimum (512MB recommended)
CPU: 1.5GHz+, 4 Cores strongly recommended due to running three simultaneous threads.
HDMI Splitter (Must be able to output 4k & 1080/720p simultaneously) Here is a good one for $25, though it breaks HDR when downscaling output 2. The goal here is one output of 4K and another output of 1080/720p.
USB3.0 HDMI Capture Card (Capable of capturing 720/1080p; delay should be 50ms or under.) I got this when it was $45. A similar one should be fine. These are untested: Panoraxy | Aliexpress (This shape/style tends to perform well.)

Setup

Software Setup:

Download the latest scripts and install all dependencies via the following commands. Be sure to watch for errors! You will need about 1GB of free space. The script can run for up to an hour.

git clone https://github.com/MCPCapital/HarmonizeProject.git
cd HarmonizeProject
sudo ./setup.sh

Hardware Setup:

Connect Video Device (PS4, FireStick, etc.) to the splitter input.
Connect an HDMI cable from the 4k output to the TV; and from Output 2 (downscaled) to the video capture card connected to your device.
Ensure your splitter's switches are set to downscale Output 2 to 1080 or 720p!

Entertainment Area Configuration:

Hue App -> Settings -> Entertainment Areas
Harmonize will use the height and the horizontal position of lights in relation to the TV. The depth/vertical position are currently ignored.
In the example below, the light on the left is to the left of the TV at the bottom of it. The light on the right is on the right side of the TV at the top of it.

First-Time Run Instructions:

If you have not set up a bridge before, the program will attempt to register you on the bridge. You will have 45 second to push the button on the bridge. Current Bug - After registering, the script will store the clientdata but fail & exit. Workaround - Simply run the script again since the data was saved.
If multiple bridges are found, you will be given the option to select one. You will have to do this every time if you have multiple bridges (for now).
If multiple entertainment areas are found, you will be given the option to select one. You can also enter this as a command line argument.

Usage

To start the program:

screen
cd harmonizeproject
./harmonize.py
Type Ctrl+A and Ctrl-D to continue running the script in the background.
To resume the terminal session use screen -r
Press ENTER to safely stop the program.

Command line arguments:

-v Display verbose output
-g # Use specific entertainment group number (#)

Configurable values within the script: (Advanced users only)

Line 237 - breadth - determines the % from the edges of the screen to use in calculations. Default is 15%. Lower values can result in less lag time, but less color accuracy.
Line 315 - time.sleep(0.01) - Determines how frequently messages are sent to the bridge. Keep in mind the rest of the function takes some time to run in addition to this sleep command. Bridge requests are capped by Philips at a rate of 60/s (1 per ~16.6ms) and the excess are dropped.
Run with sudo to give Harmonize higher priority over other CPU tasks.

Troubleshooting

"Import Error" - Ensure you have all the dependencies installed. Run through the manual dependency install instructions above.
No video input // lights are all dim gray - Run python3 ./videotest.py to see if your device (via OpenCV) can properly read the video input.
w, h, or rgbframe not defined - Increase the waiting time from 0.75 seconds - Line 330 {time.sleep(.75)} *This is a known bug (race condition).
python3-opencv installation fails - Compile from source - Follow this guide.
Many questions are answered on our Reddit release thread here. New issues should be raised on GitLab.

Contributions & License

Pull requests are encouraged and accepted! Whether you have some code changes or enhancements to the readme, feel free to open a pull request. Harmonize Project is licensed under The Creative Commons Attribution-NonCommercial 4.0 International Public License.

Development credits to Matthew C. Pilsbury (MCP Capital, LLC) and Ares N. Vlahos.

ChrisB9 / HarmonizeProject