Finding Fallen Objects Via Asynchronous Audio-Visual Integration

Chuang Gan; Yi Gu; Siyuan Zhou; Jeremy Schwartz; Seth Alter; James Traer; Dan Gutfreund; Joshua B Tenenbaum; Josh H McDermott; Antonio Torralba

doi:10.1109/CVPR52688.2022.01027

Conference proceeding

Finding Fallen Objects Via Asynchronous Audio-Visual Integration

Chuang Gan, Yi Gu, Siyuan Zhou, Jeremy Schwartz, Seth Alter, James Traer, Dan Gutfreund, Joshua B Tenenbaum, Josh H McDermott and Antonio Torralba

The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Conference Proceedings

01/01/2022

DOI: 10.1109/CVPR52688.2022.01027

View Online

Abstract

Conference Title: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Conference Start Date: 2022, June 18 Conference End Date: 2022, June 24 Conference Location: New Orleans, LA, USAThe way an object looks and sounds provide complementary reflections of its physical properties. In many settings cues from vision and audition arrive asynchronously but must be integrated, as when we hear an object dropped on the floor and then must find it. In this paper, we introduce a setting in which to study multi-modal object localization in 3D virtual environments. An object is dropped somewhere in a room. An embodied robot agent, equipped with camera and microphone, must determine what object has been dropped - and where - by combining audio and visual signals with knowledge of the underlying physics. To study this problem, we have generated a large-scale dataset - the Fallen Objects dataset - that includes 8000 instances of 30 physical object categories in 64 rooms. The dataset uses the ThreeDWorld Platform that can simulate physics-based impact sounds and complex physical interactions between objects in a photorealistic setting. As a first step toward addressing this challenge, we develop a set of embodied agent baselines, based on imitation learning, reinforcement learning, and modular planning, and perform an in-depth analysis of the challenge of this new task. This dataset is publicly available11Project page: http://fallen-object.csail.mit.edu.

Acoustics

Computer vision

Datasets

Learning

Pattern recognition

Physical properties

Virtual environments

Visual signals

Details

Title: Subtitle: Finding Fallen Objects Via Asynchronous Audio-Visual Integration
Creators: Chuang Gan
Yi Gu
Siyuan Zhou
Jeremy Schwartz
Seth Alter
James Traer
Dan Gutfreund
Joshua B Tenenbaum
Josh H McDermott
Antonio Torralba
Resource Type: Conference proceeding
Publication Details: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Conference Proceedings
DOI: 10.1109/CVPR52688.2022.01027
Publisher: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Grant note: DOI: 10.13039/100000006, name: ONR, award: N00014-18-1-2847; DOI: 10.13039/100000001, name: NSF, award: 1921501
Language: English
Date published: 01/01/2022
Academic Unit: Psychological and Brain Sciences; Iowa Neuroscience Institute
Record Identifier: 9984302602602771

Metrics

26 Record Views

13 Times Cited - Web of Science