How to modify input to accept BEV

[suzannajiwani]

Hi,
I'm an undergraduate researcher working with a lab. We would like to use your model as a baseline, but there are a few modifications we need to make. The first is the input - the input to our model is a birds-eye view of a scene, while the input to fiery is a series of camera images. Looking at Figures 2 and 3 of your paper, it seems like it would be possible to cut in after the features are projected to birds-eye view. After working with the model, however, it seems difficult to substitute the vector x with a bird-eye view. Do you have any suggestions for how this may be done?

[Anthony Hu]

Hey Suzanna! Thank you for your interest in our work. If you want to use bird's-eye view inputs, then you'll need to comment out the following lines in fiery/models/fiery.py L133:140

# Only process features from the past and present
image = image[:, :self.receptive_field].contiguous()
intrinsics = intrinsics[:, :self.receptive_field].contiguous()
extrinsics = extrinsics[:, :self.receptive_field].contiguous()
future_egomotion = future_egomotion[:, :self.receptive_field].contiguous()

# Lifting features and project to bird's-eye view
 x = self.calculate_birds_eye_view_features(image, intrinsics, extrinsics)

and set x to the bird's-eye inputs. You'll also need to modify the number of input channel of the temporal model so that it matches the dimension of the BEV inputs _C.MODEL.TEMPORAL_MODEL.START_OUT_CHANNELS = ...

Let me know if you have any other question!