Conference proceeding
Generative Modeling of Audible Shapes for Object Perception
2017 IEEE International Conference on Computer Vision (ICCV), Vol.2017-, pp.1260-1269
10/2017
DOI: 10.1109/ICCV.2017.141
Abstract
Humans infer rich knowledge of objects from both auditory and visual cues. Building a machine of such competency, however, is very challenging, due to the great difficulty in capturing large-scale, clean data of objects with both their appearance and the sound they make. In this paper, we present a novel, open-source pipeline that generates audiovisual data, purely from 3D object shapes and their physical properties. Through comparison with audio recordings and human behavioral studies, we validate the accuracy of the sounds it generates. Using this generative model, we are able to construct a synthetic audio-visual dataset, namely Sound-20K, for object perception tasks. We demonstrate that auditory and visual information play complementary roles in object perception, and further, that the representation learned on synthetic audio-visual data can transfer to real-world scenarios.
Details
- Title: Subtitle
- Generative Modeling of Audible Shapes for Object Perception
- Creators
- Zhoutong ZhangJiajun WuQiujia LiZhengjia HuangJames TraerJosh H McDermottJoshua B TenenbaumWilliam T Freeman
- Resource Type
- Conference proceeding
- Publication Details
- 2017 IEEE International Conference on Computer Vision (ICCV), Vol.2017-, pp.1260-1269
- Publisher
- IEEE
- DOI
- 10.1109/ICCV.2017.141
- ISSN
- 1550-5499
- eISSN
- 2380-7504
- Language
- English
- Date published
- 10/2017
- Academic Unit
- Iowa Neuroscience Institute; Psychological and Brain Sciences
- Record Identifier
- 9984070485602771
Metrics
10 Record Views