Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices

Dixian Zhu; Xiaodong Wu; Tianbao Yang

Back

Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices

Preprint

Open access

Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices

Dixian Zhu, Xiaodong Wu and Tianbao Yang

ArXiv.org

03/26/2022

Files and links (1)

url

https://doi.org/10.48550/arXiv.2203.14177View

Preprint (Author's original)This preprint has not been evaluated by subject experts through peer review. Preprints may undergo extensive changes and/or become peer-reviewed journal articles. Open Access

Abstract

The area under the ROC curve (AUROC) has been vigorously applied for imbalanced classification and moreover combined with deep learning techniques. However, there is no existing work that provides sound information for peers to choose appropriate deep AUROC maximization techniques. In this work, we fill this gap from three aspects. (i) We benchmark a variety of loss functions with different algorithmic choices for deep AUROC optimization problem. We study the loss functions in two categories: pairwise loss and composite loss, which includes a total of 10 loss functions. Interestingly, we find composite loss, as an innovative loss function class, shows more competitive performance than pairwise loss from both training convergence and testing generalization perspectives. Nevertheless, data with more corrupted labels favors a pairwise symmetric loss. (ii) Moreover, we benchmark and highlight the essential algorithmic choices such as positive sampling rate, regularization, normalization/activation, and optimizers. Key findings include: higher positive sampling rate is likely to be beneficial for deep AUROC maximization; different datasets favors different weights of regularizations; appropriate normalization techniques, such as sigmoid and ℓ2 score normalization, could improve model performance. (iii) For optimization aspect, we benchmark SGD-type, Momentum-type, and Adam-type optimizers for both pairwise and composite loss. Our findings show that although Adam-type method is more competitive from training perspective, but it does not outperform others from testing perspective.

Details

Title: Subtitle: Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices
Creators: Dixian Zhu
Xiaodong Wu
Tianbao Yang
Resource Type: Preprint
Publication Details: ArXiv.org
ISSN: 2331-8422
Language: English
Date posted: 03/26/2022
Academic Unit: The Iowa Institute for Biomedical Imaging; Radiation Oncology; Electrical and Computer Engineering; Computer Science
Record Identifier: 9984230416402771

Metrics

73 Record Views