ForRMByishi / Kinect_Smoothing

"Kinect Smoothing" helps you to smooth and filter the Kinect depth image as well as trajectory data

Kinect Depth Image Smoothing

• "Kinect Smoothing" helps you to smooth and filter the Kinect depth image and trajectory data.

How to Use it

• Installation

pip install -r requirements.txt
git clone https://github.com/intelligent-control-lab/Kinect_Smoothing.git

• Running

  see example.ipynb

Features

1 . Depth Image Smoothing

• Hole Filling Filter

  In the original Kinect depth image, there are many invalid pixels (they appear black on the image and are registered as 0's). This function helps you to fill the invalid values with based on the valid pixels in the vicinity. The methods for hole filling are as follows:

  • min: Fill in with the neighboring minimum valid value
  • max: Fill in with the neighboring maximum valid value, refer to A computationally efficient denoising and hole-filling method for depth image enhancement
  • mean: Fill in with the average of all neighboring valid values
  • mode: Fill in with the mode of neighboring valid values, refer to Smoothing Kinect Depth Frames in Real-Time
  • fmi: Fast Matching Inpainting, refer to An Image Inpainting Technique Based on the Fast Marching Method
  • ns: Fluid Dynamics Inpainting, refer to Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting

• Denoising Filter

  After pixel filling, Denoising filters can be used to improve the resolution of the depth image. The methods for denoising filling are as follows:

  • modeling: filter with Kinect V2 noise model, refer to Modeling Kinect Sensor Noise for Improved 3D Reconstruction and Tracking
  • modeling_pf: another Kinect V2 noise modeling by Peter Fankhauser Kinect v2 for Mobile Robot Navigation: Evaluation and Modeling
  • anisotropic: smoothing with anisotropic filtering Scale-space and edge detection using anisotropic diffusion
  • gaussian: smoothing with Gaussian filtering

2 . Trajectory Smoothing

• Crop Filter:

  The x, y coordinates of the trajectory were captured by some object detection algorithms (e.g. Openpose). Sometimes the object will be positioned on the background, and the depth coordinates might register as invalid values on the Kinect. The Crop-Filter crops the invalid value and run some interpolation methods to replace it. The methods for invalid value replacement are as follows:

  • convenient interpolation methods: such as "zero","linear","slinear","quadratic","cubic","previous","next","nearest".
  • pchip: PCHIP 1-d monotonic cubic interpolation, refer Monotone Piecewise Cubic Interpolation
  • akima: Akima 1D Interpolator, refer to A new method of interpolation and smooth curve fitting based on local procedures

• Smooth Filter:

  Smooth the data with a specific filter, which is effectively reducing the anomaly in the trajectory series. The methods for smoothing filter are as follows:

  • kalman: smooth the signal with Kalman filter, refer to Fundamentals of Kalman Filtering
  • wiener: smooth the signal with Wiener filter
  • median: smooth the signal with median filter
  • moving_average: smooth the signal with moving average filter
  • exponential_moving_average: smooth the signal with exponential moving average filter

• Motion Sampler:

  Remove the non-moving part of the trajectory. (Just keep the significant movements)

3 . Real World Coordinate Calculation

• Coordinate Calculator

  Transform the pixel level coordinate of Kinect to the real world coordinate.