Conference proceeding
Deep Incremental RNN for Learning Sequential Data: A Lyapunov Stable Dynamical System
2021 IEEE International Conference on Data Mining (ICDM), Vol.2021-, pp.966-975
12/2021
DOI: 10.1109/ICDM51629.2021.00108
Abstract
With the recent advances in mobile sensing technologies, large amounts of sequential data are collected, such as vehicle GPS records, stock prices, sensor data from air quality detectors. Recurrent neural networks (RNNs) have been studied extensively to learn complex patterns for sequential data, with applicatons in natural language processing for sentence prediction/completion, human activity recognition for predicting or classifying human activities. However, there are many practical issues when training RNNs, e.g., vanishing and exploding gradients often occur due to the repeatability of network weights, etc. In this paper, we study the training stability in deep recurrent neural networks (RNNs), and propose a novel network, namely, deep incremental RNN (DIRNN). In contrast to the literature, we prove that DIRNN is essentially a Lyapunov stable dynamical system where there is no vanishing or exploding gradient in training. To demonstrate the applicability in practice, we also propose a novel implementation, namely TinyRNN, that sparsifies the transition matrices in DIRNN using weighted random permutations to reduce the model sizes. We evaluate our approach on seven benchmark datasets, and achieve state-of-the-art results. Demo code is provided in the supplementary file.
Details
- Title: Subtitle
- Deep Incremental RNN for Learning Sequential Data: A Lyapunov Stable Dynamical System
- Creators
- Ziming Zhang - Worcester Polytechnic InstituteGuojun Wu - Worcester Polytechnic InstituteYanhua Li - Worcester Polytechnic InstituteYun Yue - Worcester Polytechnic InstituteXun Zhou - University of Iowa
- Resource Type
- Conference proceeding
- Publication Details
- 2021 IEEE International Conference on Data Mining (ICDM), Vol.2021-, pp.966-975
- Publisher
- IEEE
- DOI
- 10.1109/ICDM51629.2021.00108
- ISSN
- 1550-4786
- eISSN
- 2374-8486
- Grant note
- U.S. Department of Transportation (10.13039/100000140)
- Language
- English
- Date published
- 12/2021
- Academic Unit
- Business Analytics
- Record Identifier
- 9984380510802771
Metrics
2 Record Views