haoranD / ASPIRe

[CVPR 2024] HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding

HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding (CVPR 2024)

HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding

Trong-Thuan Nguyen, Pha Nguyen, Khoa Luu

Abstract

Visual interactivity understanding within visual scenes presents a significant challenge in computer vision. Existing methods focus on complex interactivities while leveraging a simple relationship model. These methods, however, struggle with a diversity of appearance, situation, position, interaction, and relation in videos. This limitation hinders the ability to fully comprehend the interplay within the complex visual dynamics of subjects. In this paper, we delve into interactivities understanding within visual content by deriving scene graph representations from dense interactivities among humans and objects. To achieve this goal, we first present a new dataset containing Appearance-Situation-Position-Interaction-Relation predicates, named ASPIRe, offering an extensive collection of videos marked by a wide range of interactivities. Then, we propose a new approach named Hierarchical Interlacement Graph (HIG), which leverages a unified layer and graph within a hierarchical structure to provide deep insights into scene changes across five distinct tasks. Our approach demonstrates superior performance to other methods through extensive experiments conducted in various scenarios.

Introduction

We introduce the new ASPIRe dataset to Visual Interactivity Understanding. The diversity of the ASPIRe dataset is showcased through its wide range of scenes and settings, distributed in seven scenarios.

Examples of annotations found on the ASPIRe dataset.

teaser-aspire.mp4

Annotations

v1.0:

• The category name, bbox, segmentation and track_id compatible with TAO dataset.
  • Training set: [Train annotations],
  • Validation set: [Validation annotations],

Licensing:

The annotations of ASPIRe and the original source videos are released under a CC BY-NC-SA 3.0 license per their creators. See motchallenge.net for details.

This page was built using the Academic Project Page Template.
This website is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.