Abstract

Weakly-supervised temporal action localization aims to recognize and localize action segments in untrimmed videos given only video-level action labels for training. Without the boundary information of action segments, existing methods mostly rely on multiple instance learning (MIL), where the predictions of unlabeled instances (i.e., video snippets) are supervised by classifying labeled bags (i.e., untrimmed videos). However, this formulation typically treats snippets in a video as independent instances, ignoring the underlying temporal structures within and across action segments. To address this problem, we propose ASM-Loc, a novel WTAL framework that enables explicit, action-aware segment modeling beyond standard MIL-based methods. Our framework entails three segment-centric components: (i) dynamic segment sampling for compensating the contribution of short actions; (ii) intra- and inter-segment attention for modeling action dynamics and capturing temporal dependencies; (iii) pseudo instance-level supervision for improving action boundary prediction. Furthermore, a multi-step refinement strategy is proposed to progressively improve action proposals along the model training process. Extensive experiments on THUMOS-14 and ActivityNet-v1.3 demonstrate the effectiveness of our approach, establishing new state of the art on both datasets.

Motivation

There are two common failures in existing MIL-based methods: 1) missed detection of short action segments; 2) incomplete or over-complete action segments. These MIL-based methods treat snippets in a video as independent instances, where their underlying temporal structures are neglected in either the feature modeling or prediction stage. Our ASM-Loc leverages the action proposals as well as the proposed segment-centric modules to address the common failures in existing MIL-based methods.

Model overview

Our ASM-Loc is a novel WTAL framework with explicit, action-aware segment modeling modules Leverage action proposals generated by the standard MIL-based methods.

• The Dynamic Segment Sampling module dynamically up-samples proposals with short durations.
• The Intra- and Inter-segment attention modules capture temporal structures within & across action segments.
• The Pseudo Instance-level Loss improves boundary prediction by leveraging finer-grained supervision.
• A multi-step refinement process to improve the quality of action proposals and the final localization result.

Experiments

THUMOS-14 Dataset

ActivityNet-v1.3 Dataset

Qualitative

The website template was borrowed from Ben Mildenhall.