Book chapter
Accelerating Deep Learning with Millions of Classes
Computer Vision – ECCV 2020, pp.711-726
Lecture Notes in Computer Science, Springer International Publishing
11/03/2020
DOI: 10.1007/978-3-030-58592-1_42
Abstract
Deep learning has achieved remarkable success in many classification tasks because of its great power of representation learning for complex data. However, it remains challenging when extending to classification tasks with millions of classes. Previous studies are focused on solving this problem in a distributed fashion or using a sampling-based approach to reduce the computational cost caused by the softmax layer. However, these approaches still need high GPU memory in order to work with large models and it is non-trivial to extend them to parallel settings. To address these issues, we propose an efficient training framework to handle extreme classification tasks based on Random Projection. The key idea is that we first train a slimmed model with a random projected softmax classifier and then we recover it to the original classifier. We also show a theoretical guarantee that this recovered classifier can approximate the original classifier with a small error. Later, we extend our framework to parallel settings by adopting a communication reduction technique. In our experiments, we demonstrate that the proposed framework is able to train deep learning models with millions of classes and achieve above 10×\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$10{\times }$$\end{document} speedup compared to existing approaches.
Details
- Title: Subtitle
- Accelerating Deep Learning with Millions of Classes
- Creators
- Zhuoning Yuan - University of IowaZhishuai Guo - University of IowaXiaotian Yu - Shenzhen, ChinaXiaoyu Wang - Chinese University of Hong KongTianbao Yang - University of Iowa
- Resource Type
- Book chapter
- Publication Details
- Computer Vision – ECCV 2020, pp.711-726
- Publisher
- Springer International Publishing; Cham
- Series
- Lecture Notes in Computer Science
- DOI
- 10.1007/978-3-030-58592-1_42
- eISSN
- 1611-3349
- ISSN
- 0302-9743
- Language
- English
- Date published
- 11/03/2020
- Academic Unit
- Computer Science
- Record Identifier
- 9984259493702771
Metrics
4 Record Views