Literature by the same author
plus at Google Scholar

Bibliografische Daten exportieren
 

Sensitivity-Based Layer Insertion for Residual and Feedforward Neural Networks

Title data

Herberg, Evelyn ; Herzog, Roland ; Köhne, Frederik ; Kreis, Leonie ; Schiela, Anton:
Sensitivity-Based Layer Insertion for Residual and Feedforward Neural Networks.
Bayreuth ; Heidelberg , 2023 . - 15 p.
DOI: https://doi.org/10.48550/arXiv.2311.15995

Official URL: Volltext

Project information

Project title:
Project's official title
Project's id
Multilevel Architectures and Algorithms in Deep Learning
464103607

Project financing: Deutsche Forschungsgemeinschaft

Abstract in another language

The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic method to insert new layers during the training process, which eliminates the need to choose a fixed network size before training. Our technique borrows techniques from constrained optimization and is based on first-order sensitivity information of the objective with respect to the virtual parameters that additional layers, if inserted, would offer. We consider fully connected feedforward networks with selected activation functions as well as residual neural networks. In numerical experiments, the proposed sensitivity-based layer insertion technique exhibits improved training decay, compared to not inserting the layer. Furthermore, the computational effort is reduced in comparison to inserting the layer from the beginning. The code is available at https://github.com/LeonieKreis/layer_insertion_sensitivity_based.

Further data

Item Type: Preprint, postprint
Refereed: Yes
Keywords: constructive neural networks; layer insertion; sensitivity analysis; network architecture; deep learning
Institutions of the University: Faculties
Faculties > Faculty of Mathematics, Physics und Computer Science
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Mathematics
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Mathematics > Chair Mathematics V (Applied Mathematics)
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Mathematics > Chair Applied Mathematics
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Mathematics > Chair Applied Mathematics > Chair Applied Mathematics - Univ.-Prof. Dr. Anton Schiela
Profile Fields
Profile Fields > Advanced Fields
Profile Fields > Advanced Fields > Nonlinear Dynamics
Result of work at the UBT: Yes
DDC Subjects: 500 Science > 510 Mathematics
Date Deposited: 30 Nov 2023 06:16
Last Modified: 30 Nov 2023 06:16
URI: https://eref.uni-bayreuth.de/id/eprint/87948