A fun project to separate persons from background in a Video4Linux stream. The result is provided as virtualVideo4Linux device and hence can be used to bring "virtual background" support to any video-streaming app. And all of that without a green screen.
The idea (conceptionally and implementation-wise) is taken from Benjamin Elder's amazing blog post "Open Source Virtual Background". There are a few key differences though:
- No NodeJS helper application - (almost) all in Python
- FFmpeg is used for collecting and playing back frames
What stays the same?
- v4l2loopback to provide the modified stream as video device
- BodyPix model to provide the keying feature itself
- OpenCV for some image manipulations
- Python 3.7
- FFmpeg with v4l support
- Tensorflow
- OpenCV 4
- A BodyPix model in Tensorflow's "frozen graph" format (ProtoBuf)
The model can be downloaded in TensorflowJS format from Google's servers:
curl -O https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/mobilenet/float/075/model-stride16.json
curl -O https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/mobilenet/float/075/group1-shard1of2.bin
curl -O https://storage.googleapis.com/tfjs-models/savedmodel/bodypix/mobilenet/float/075/group1-shard2of2.binOne can then use tfjs-to-tf to convert it into a Tensorflow-understandable model.