Image Preprocessing Strategies

[Moritz Reuss]

Hi,

thanks for sharing this cool project!
I have a general question regarding the different preprocessing steps of the images for training in the different environments. I noticed, that in MW you are using CenterCrop + RandomCrop, while in all the other environments only CenterCrop is applied.
Could you elaborate on this choice for MW? Is MW easier to overfit?

Thanks in advance!

[Po-Chen Ko]

Hello, thanks for reaching out!
I want to clarify that the results reported in our paper were achieved with CenterCrop only. This holds for the MW, iTHOR, and all other experiments we conducted. In this repo, we incorporated RandomCrop in the MW preprocessing step because our internal findings (which were not detailed in the paper as they were somewhat beyond its scope) indicate a noticeable improvement in task performance with this addition. Specifically, each task showed a consistent improvement, and the overall performance increased by roughly 2%. - A free improvement!

The reason why we did not apply the RandomCrop is simply because we haven't had a chance to experiment with RandomCrop with other settings. Hope this information helps!

[Moritz Reuss]

@kbkbowo Thank you for the detailed explanation, that is really helpfull! I implemented the video model in another environment and given your code it was really easy, thanks for releasing such a clean and easy to use code-base!

I got one general follow-up question regarding the choice of hyperparameters for sim and real: I guess the real environment is the most challenging one for the video diffusion model to learn, given its high number of distractions and objects in every scene. So I was suprised to see, that you used a model with smaller number of parameters and lower resolution for this task. Could you elaborate on this choice? Thanks:)