ld-ing / tcfpn-isba

Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment, CVPR 2018.

Temporal Convolutional Feature Pyramid Network (TCFPN) &
Iterative Soft Boundary Assignment (ISBA)

Introduction

In this work, we address the task of weakly-supervised human action segmentation in long, untrimmed videos. We propose a novel action modeling framework, which consists of a new temporal convolutional network, named Temporal Convolutional Feature Pyramid Network (TCFPN), for predicting frame-wise action labels, and a novel training strategy for weakly-supervised sequence modeling, named Iterative Soft Boundary Assignment (ISBA), to align action sequences and update the network in an iterative fashion. Details can be found in Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (Li Ding & Chenliang Xu, CVPR '18)

This repo includes Keras + Tensorflow implementation on Breakfast dataset. Tested with Ubuntu 16.04 + Python 2.7.

---

Overview

• train_spv.py - supervised training and testing using ED-TCN / TCFPN

• train_weak.py - weakly-supervised training and testing using ED-TCN / TCFPN + ISBA

• utils/... - model definitions, metrics, utils

• data/... - intruction to get the data

---

Quick Start

Please follow the instruction in data/README to obtain the data, then run

python train_spv.py for supervised experiments on Breakfast dataset

or

python train_weak.py for weakly-supervised experiments on Breakfast dataset.

---

Citing

If you find TCFPN / ISBA useful in your research, please consider citing:

@InProceedings{Ding_2018_CVPR,
author = {Ding, Li and Xu, Chenliang},
title = {Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2018}
}