Semantic Fusion Augmentation and Semantic Boundary Detection: A Novel Approach to Multi-Target Video Moment Retrieval

This is the official implementation of the paper Semantic Fusion Augmentation and Semantic Boundary Detection: A Novel Approach to Multi-Target Video Moment Retrieval (WACV 2024).

Pyhton Environments

Install python packages.
```
$ pip install -r requirements.txt
```
(Optional) Boost mAP calculation.
```
$ python setup.py install
```

Datasets

The datasets can be downloaded from our OneDrive.

The folder structure should be like this:

./data
├── ActivityNet
│   ├── C3D
│   │   └── activitynet_v1-3_c3d.hdf5
│   ├── I3D
│   │   ├── v_00Dk03Jr70M.npy
│   │   ├── v_00KMCm2oGhk.npy
│   │   ├── ...
|   |   └── ...
│   ├── multi_test.json
│   ├── test.json
│   ├── train.json
│   └── val.json
├── CharadesSTA
│   ├── VGG
│   │   └── vgg_rgb_features.hdf5
│   ├── C3D
│   │   └── Charades_C3D.hdf5
│   ├── I3D
│   │   ├── 001YG.npy
│   │   ├── 003WS.npy
│   │   ├── ...
|   |   └── ...
│   ├── multi_test.json
│   ├── test.json
│   └── train.json
└── QVHighlights
    ├── features
    │   ├── clip_features
    │   ├── clip_text_features
    │   └── slowfast_features
    ├── test.json
    ├── train.json
    └── val.json

Training

All our training configuration files are in the ./configs folder. The training command is as follows:

Single GPU training.

$ python main.py --config path/to/config.json --logdir path/to/log/dir

For example, to train the model on CharadesSTA dataset with VGG backbone:

$ python main.py --config ./configs/charades-VGG.json --logdir ./logs/charades-VGG-log

Multi-GPU training. For example, to train the model on CharadesSTA dataset with VGG backbone on 4 GPUs:

$ CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --config ./configs/charades-VGG.json --logdir ./logs/charades-VGG-log

Pretrained Models

The pretrained models used in our paper can be downloaded from OneDrive.

It is recommended to put the pretrained models in the ./logs folder.

./logs
├── activity-C3D-log
│   ├── ...
├── activity-I3D-log
│   ├── ...
├── charades-C3D-log
│   ├── ...
├── charades-I3D-log
│   ├── ...
├── charades-VGG-log
│   ├── best.pth
│   └── config.json
└── qv-log
    ├── ...

Reproduce the results in our paper. Take the CharadesSTA dataset as an example:

$ python main.py --test_only --config ./logs/charades-VGG-log/config.json  --logdir ./logs/charades-VGG-log

Citation

If you find this code useful for your research, please cite our paper:

@InProceedings{Huang_2024_WACV,
    author    = {Cheng Huang, Yi-Lun Wu, Hong-Han Shuai, Ching-Chun Huang},
    title     = {Semantic Fusion Augmentation and Semantic Boundary Detection: A Novel Approach to Multi-Target Video Moment Retrieval},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2024}
}

basiclab / SFABD