Conference proceeding
Contrastively Disentangled Sequential Variational Autoencoder
Advances in Neural Information Processing Systems 34 (NeurIPS 2021), Vol.34
Advances in Neural Information Processing Systems
01/01/2021
Abstract
Self-supervised disentangled representation learning is a critical task in sequence modeling. The learnt representations contribute to better model interpretability as well as the data generation, and improve the sample efficiency for downstream tasks. We propose a novel sequence representation learning method, named Contrastively Disentangled Sequential Variational Autoencoder (C-DSVAE), to extract and separate the static (time-invariant) and dynamic (time-variant) factors in the latent space. Different from previous sequential variational autoencoder methods, we use a novel evidence lower bound which maximizes the mutual information between the input and the latent factors, while penalizes the mutual information between the static and dynamic factors. We leverage contrastive estimations of the mutual information terms in training, together with simple yet effective augmentation techniques, to introduce additional inductive biases. Our experiments show that C-DSVAE significantly outperforms the previous state-of-the-art methods on multiple metrics.
Details
- Title: Subtitle
- Contrastively Disentangled Sequential Variational Autoencoder
- Creators
- Junwen Bai - Cornell UniversityWeiran Wang - Google, Mountain View, CA 94043 USACarla Gomes - Cornell University
- Contributors
- M Ranzato (Editor)A Beygelzimer (Editor)Y Dauphin (Editor)P S Liang (Editor)J W Vaughan (Editor)
- Resource Type
- Conference proceeding
- Publication Details
- Advances in Neural Information Processing Systems 34 (NeurIPS 2021), Vol.34
- Publisher
- NeurIPS Proceedings
- Series
- Advances in Neural Information Processing Systems
- ISSN
- 1049-5258
- Number of pages
- 14
- Language
- English
- Date published
- 01/01/2021
- Academic Unit
- Computer Science
- Record Identifier
- 9984696577902771
Metrics
1 Record Views