The whole video processing is a pipeline. There are three different nodes on pipeline: source node, sink node and process node with input and output. An OpenGL ES texture is passing between nodes. That is wrapped as FrameBuffer.
Key Helper Class:
- VideoContext Create an EAGLContext and DispatchQueue. Connect EAGLContext and DispatchQueue as a pair. All operations about this EAGLContext will be send to this DispatchQueue. This DispatchQueue is a serial queue. That avoid problems caused by concurrent operations on EAGLContext.
- ShaderProgram A wrapper of OpenGL ES shader program. Writted in C++. Bridge into Swift use Objective-C.
- FrameBuffer Briefly talk. This a wrapper of OpenGL ES texture. Detailly talk. Use a CVPixelBuffer as data source when create a CVOpenGLESTexture. Also create an OpenGL Frame Buffer. Bind OpenGL ES texture with OpenGL ES frame buffer.
- FrameBufferCache Cache FrameBuffer base on texture size.
Node on video processing pipeline:
- VideoOutput Video output node. Send texture to multiple targets.
- VideoInput Video input node. Receive one texture.
- VideoCamera Source node. Caputre camera video frame. Turn it into texture. Send out.
- VideoFilterProcessor Process node. Receive one texture. Go through video filters chain. Send Out.
- VideoView Sink node. Receive one texture. Render on CAEAGLLayer.
First. Create an AVCaptureSession like above graph. Implement AVCaptureVideoDataOutputSampleBufferDelegate to get camera video data as CMSampleBuffer.
After that. Create an OpenGL ES pipeline to turn video data into OpenGL ES texture. Also with orientation change.
Finally. OpenGL ES render into output texture. Use FrameBuffer to wrap it. Inside FrameBuffer. It also use a CVPixelBuffer as data source when create a CVOpenGLESTexture. But this CVPixelBuffer is empty for right now. Before OpenGL ES render. Use FrameBufferCache to get FrameBuffer. Active FrameBuffer to bind OpenGL ES frame buffer as current frame buffer. When create FrameBuffer, OpenGL ES texture already bind with OpenGL ES frame buffer. So this can guarantee OpenGL ES render into this texture.
Majorly divide into three parts: VideoFilterProcessor (Swift). Bridge (Objective-C). Core (C++). Let`s talk about them from bottom to top:
You should understand a video effect: run an OpenGL ES shader program to do image processing. Shader program also have inputs and outputs. Outputs are mainly output texture. Inputs are mainly input texture and other parameters to control shader program.
VideoEffectProcessor is an interface to control video effects. There is a video effects chain inside it. Receive a texture. Go through video effects chain. Send out a texture. Methods to control video effect with a name parameter. Every video effect have an unique name. Use this name to find specific video effect in video effects chain. That lead to an issue of cannot add same video effect more than once. I will solve this issue when I have time. 🙃
VideoEffect is base class of video effect. Be responsible for loading OpenGL ES shader program, feeding shader program with parameters need to run, actually running shader program. Send out processed texture. Base on characteristics of video effect, there are different subclasses.
Mainly base classes:
- VideoTwoPassEffect Run two shader programs.
- VideoTwoPassTextureSamplingEffect Inherited from VideoTwoPassEffect. Run two shader programs. First run with vertically sampling. Second run with horizontally sampling.
- Video3x3TextureSamplingEffect Sampling 8 point around center point. Plus center point. Sampling 9 points.
- Video3x3ConvolutionEffect Inherited from Video3x3TextureSamplingEffect. Get 3x3 sampling points. Multiply 3x3 matrix. Assignment result to center point. This is convolution.
- VideoTwoInputEffect Send two textures to one same shader program. Blend two textures on one rectangle.
- VideoCompositionEffect One same shader program run twice. First run with a video texture as input. Second run with an image texture as input. So composite results of two runs. Implement water mark, animated sticker and text drawing.
5 different kinds of subclasses:
- Color Processing
- VideoBrightnessEffect Adjust brightness.
- VideoExposureEffect Adjust exposure.
- VideoContrastEffect Adjust contrast.
- VideoSaturationEffect Adjust saturation.
- VideoGammaEffect Adjust gamma.
- VideoLevelsEffect Photoshop-like levels adjustment.
- VideoColorMatrixEffect Color of a pixel is 1x4 column vector. Multiply a 4x4 matrix to get a new color.
- VideoSepiaFilter Use VideoColorMatrixEffect implement a sepia effect.
- VideoRGBEffect Adjust red, green and blue seperatly.
- VideoHueEffect Adjust hue.
- Gray Scale Turn into gray.
- Image Processing
- VideoTransformEffect Transform image with scale, rotate and translate. Both 2D and 3D transforms are allowed.
- VideoCropFilter Crop image. It`s not really crop. Output texture size is not change. Only change visible are of this image.
- VideoGaussianBlurEffect Gaussian blur.
- VideoBoxBlurEffect Box blur.
- VideoSobelEdgeDetectionEffect Sobel edge detection.
- VideoSharpenEffect Sharpen.
- VideoBilateralEffect Bilateral.
- Blend
- Add Blend
- Alpha Blend
- VideoMaskEffect Provide an image texture as mask for video texture. White will pass. Black will be deleted.
- Multiply Blend
- Screen Blend
- Overlay Blend
- Soft Light Blend
- Hard Light Blend
- Composition
- VideoWaterMaskEffect Provide an image texture as water mask.
- VideoAnimatedStickerEffect Provide a series of images sequence to do animated sticker.
- VideoTextEffect Add text. iOS draw a text into an image. Send it to C++.
- Effect
- VideoEmbossFilter Emboss effect.
- VideoToonEffect Cartoon effect.
- VideoSketchEffect Sketch effect.
- VideoMosaicEffect Mosaic effect.
One VideoFilter to one VideoEffect. As before said, VideoEffect do heavy work to run shader program. VideoFilter is just data model. VideoEffectProcessorObject contain VideoEffectProcessor. Through VideoEffectProcessorObject, send VideoFilter to core part. This way can control video effects. But have to write a lots of redundant code. The essense is to exchange data between Objective-C layer and C++ layer. So just declare data exchange protocol. Use Protocol Buffers to auto generate codes. I will solve this issue when I have time. 🙃
VideoFilterProcessor is a node on video processing pipeline. Receive one texture. Go through video filters chain. Send Out. VideoFilterProcessor also contain VideoEffectProcessorObject to control video effects.
Receive one texture. Use OpenGL ES render into CAEAGLLayer. As before said, bind OpenGL ES frame buffer to texture. So it can render into texture. At here, bind OpenGL ES frame buffer to render buffer. Render buffer be connected with CAEAGLLayer. This can guarantee OpenGL ES render into CAEAGLLayer.