ORB-SLAM3 with Imu does not work

[Artur Podtikhov]

Question

What's your question?

I'm trying to run ORB-SLAM3 on the data received from Airsim. For the case of using 1 or 2 cameras, everything worked out. There are problems when using Imu. The trajectory is built completely incorrectly, and the local map is also lost too often.

Include context on what you are trying to achieve

I take images from 2 cameras with a resolution of 600x400 pixels with a frequency of 20 FPS. And Imu with a frequency of 200 Hz.

Context details

My settings.json:

{
  "SettingsVersion": 1.2,

  "SimMode": "Car",
  "RecordUIVisible": true,
  "LogMessagesVisible": true,
  "ClockSpeed": 0.1,
  "SpeedUnitFactor": 3.6,
  "SpeedUnitLabel": "km/h",
  "ViewMode": "",

  "SubWindows": [
    {
      "WindowID": 0,
      "ImageType": 0,
      "CameraName": "front_left",
      "Visible": true
    },
    {
      "WindowID": 1,
      "ImageType": 0,
      "CameraName": "front_right",
      "Visible": true
    }
  ],
  "Vehicles": {
    "EgoCar": {
      "VehicleType": "PhysXCar",
      "AutoCreate": true,
      "X": 0,
      "Y": 0,
      "Z": 0,
      "Pitch": 0,
      "Roll": 0,
      "Yaw": 0,
      "Cameras": {
        "front_left": {
          "CaptureSettings": [
            {
              "ImageType": 0,
              "Width": 600,
              "Height": 400,
              "FOV_Degrees": 90

            }
          ],
          "X": 0,
          "Y": -0.26,
          "Z": -1.9,
          "Pitch": 0,
          "Roll": 0,
          "Yaw": 0
        },
        "front_right": {
          "CaptureSettings": [
            {
              "ImageType": 0,
              "Width": 600,
              "Height": 400,
              "FOV_Degrees": 90

            }
          ],
          "X": 0,
          "Y": 0.26,
          "Z": -1.9,
          "Pitch": 0,
          "Roll": 0,
          "Yaw": 0
        }
      },

      "Sensors": {
        "Gps": {
          "SensorType": 3,
          "Enabled": true
        },
        "Imu": {
          "SensorType": 2,
          "Enabled": true
        }
      }
    }
  }
}

ORB-SLAM configuration file:

%YAML:1.0

#--------------------------------------------------------------------------------------------
# System config
#--------------------------------------------------------------------------------------------

# When the variables are commented, the system doesn't load a previous session or not store the current one

# If the LoadFile doesn't exist, the system give a message and create a new Atlas from scratch
#System.LoadAtlasFromFile: "Session_MH01_MH02_MH03_Stereo60_Pseudo"

# The store file is created from the current session, if a file with the same name exists it is deleted
#System.SaveAtlasToFile: "Session_MH01_MH02_MH03_Stereo60_Pseudo"

#--------------------------------------------------------------------------------------------
# Camera Parameters. Adjust them!
#--------------------------------------------------------------------------------------------
File.version: "1.0"

Camera.type: "PinHole"

# Camera calibration and distortion parameters (OpenCV) 
Camera1.fx: 300.0
Camera1.fy: 300.0
Camera1.cx: 300.0
Camera1.cy: 200.0

Camera1.k1: 0.0
Camera1.k2: 0.0
Camera1.p1: 0.0
Camera1.p2: 0.0

Camera2.fx: 300.0
Camera2.fy: 300.0
Camera2.cx: 300.0
Camera2.cy: 200.0

Camera2.k1: 0.0
Camera2.k2: 0.0
Camera2.p1: 0.0
Camera2.p2: 0.0

Camera.width: 600
Camera.height: 400

Camera.fps: 20
Camera.RGB: 1

# Close/Far threshold. Baseline times.
Stereo.ThDepth: 50.0
Stereo.b: 0.52

Stereo.T_c1_c2: !!opencv-matrix
  rows: 4
  cols: 4
  dt: f
  data: [ 1.0,   0.0,   0.0,   0.52,
            0.0,   1.0,   0.0,   0.0,
            0.0,  0.0,   1.0,   0.0,
            0.0,0.0,0.0,1.0]

# Transformation from camera 0 to body-frame (imu)
IMU.T_b_c1: !!opencv-matrix
  rows: 4
  cols: 4
  dt: f
  data: [ 0.0, 0.0, 1.0, 0.26,
          1.0, 0.0, 0.0, 0.0,
          0.0, 1.0, 0.0, 1.9,
          0.0, 0.0, 0.0, 1.0]
# IMU noise
IMU.NoiseGyro: 8.7266462e-5 # 1.6968e-04
IMU.NoiseAcc: 0.002353596 # 2.0000e-3
IMU.GyroWalk: 9.9735023e-7
IMU.AccWalk: 1.2481827e-5 # 3.0000e-3
IMU.Frequency: 200.0

#--------------------------------------------------------------------------------------------
# ORB Parameters
#--------------------------------------------------------------------------------------------

# ORB Extractor: Number of features per image
ORBextractor.nFeatures: 1200

# ORB Extractor: Scale factor between levels in the scale pyramid 	
ORBextractor.scaleFactor: 1.2

# ORB Extractor: Number of levels in the scale pyramid	
ORBextractor.nLevels: 8

# ORB Extractor: Fast threshold
# Image is divided in a grid. At each cell FAST are extracted imposing a minimum response.
# Firstly we impose iniThFAST. If no corners are detected we impose a lower value minThFAST
# You can lower these values if your images have low contrast			
ORBextractor.iniThFAST: 20
ORBextractor.minThFAST: 7

#--------------------------------------------------------------------------------------------
# Viewer Parameters
#--------------------------------------------------------------------------------------------
Viewer.KeyFrameSize: 0.05
Viewer.KeyFrameLineWidth: 1.0
Viewer.GraphLineWidth: 0.9
Viewer.PointSize: 2.0
Viewer.CameraSize: 0.08
Viewer.CameraLineWidth: 3.0
Viewer.ViewpointX: 0.0
Viewer.ViewpointY: -0.7
Viewer.ViewpointZ: -1.8
Viewer.ViewpointF: 500.0
Viewer.imageViewScale: 1.0

Include details of what you already did to find answers

I tried to change the transformation matrix, but without success.

[Baoyi Cui]

Did you check whether the camera's topic really reaches 20 hz? I got a similar problem when I run vins-fusion with AirSim. I just got 2 hz