Learning Video Representations from Textual Web Supervision

Abstract

Videos on the Internet are paired with pieces of text, such as titles and descriptions. This text typically describes the most important content in the video, such as the objects in the scene and the actions being performed. Based on this observation, we propose to use text as a method for learning video representations. To accomplish this, we propose a data collection process and use it to collect 70M video clips shared publicly on the Internet, and we then train a model to pair each video with its associated text. We evaluate the model on several down-stream action recognition tasks, including Kinetics, HMDB-51, and UCF-101. We find that this approach is an effective method of pre-training video representations. Specifically, it outperforms all existing methods for self-supervised and cross-modal video representation learning.

Full paper on arXiv

Details

arXiv identifier

2007.14937

Published

2020-07-29

Authors

Jonathan C. Stroud, Zhichao Lu, Chen Sun, Jia Deng, Rahul Sukthankar, Cordelia Schmid, David A. Ross

Models That Cite This Paper

• Described byvideoprism-base-f16r288
• Described byvideoprism-lvt-large-f8r288
• Described byvideoprism-lvt-base-f16r288