Literature by the same author
plus at Google Scholar

Bibliografische Daten exportieren
 

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

Title data

Köhne, Frederik ; Kreis, Leonie ; Schiela, Anton ; Herzog, Roland:
Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent.
Bayreuth ; Heidelberg , 2023 . - 31 p.
DOI: https://doi.org/10.48550/arXiv.2311.16956

Official URL: Volltext

Project information

Project title:
Project's official title
Project's id
Multilevel Architectures and Algorithms in Deep Learning
464103607

Project financing: Deutsche Forschungsgemeinschaft

Abstract in another language

This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable --- the Lipschitz constant for gradients and a concept of the local variance in search directions.
Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties when applied to quadratic problems and exhibits truly problem adaptive behavior on classical image classification tasks.
Our framework enables the potential inclusion of a preconditioner, thereby enabling the implementation of adaptive step sizes for stochastic second-order optimization methods.

Further data

Item Type: Preprint, postprint
Refereed: Yes
Keywords: stochastic gradient descent; adaptive learning rates; preconditioning; quadratic problems
Institutions of the University: Faculties
Faculties > Faculty of Mathematics, Physics und Computer Science
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Mathematics
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Mathematics > Chair Mathematics V (Applied Mathematics)
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Mathematics > Chair Applied Mathematics
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Mathematics > Chair Applied Mathematics > Chair Applied Mathematics - Univ.-Prof. Dr. Anton Schiela
Profile Fields
Profile Fields > Advanced Fields
Profile Fields > Advanced Fields > Nonlinear Dynamics
Result of work at the UBT: Yes
DDC Subjects: 500 Science > 510 Mathematics
Date Deposited: 30 Nov 2023 06:13
Last Modified: 30 Nov 2023 06:13
URI: https://eref.uni-bayreuth.de/id/eprint/87944