bei Google ScholarTitelangaben
Herberg, Evelyn ; Herzog, Roland ; Köhne, Frederik ; Kreis, Leonie ; Schiela, Anton:
Sensitivity-Based Layer Insertion for Residual and Feedforward Neural Networks.
Bayreuth ; Heidelberg
,
2023
. - 15 S.
DOI: https://doi.org/10.48550/arXiv.2311.15995
Volltext
Angaben zu Projekten
| Projekttitel: |
Offizieller Projekttitel Projekt-ID Multilevel Architectures and Algorithms in Deep Learning 464103607 |
|---|---|
| Projektfinanzierung: |
Deutsche Forschungsgemeinschaft |
Abstract
The training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic method to insert new layers during the training process, which eliminates the need to choose a fixed network size before training. Our technique borrows techniques from constrained optimization and is based on first-order sensitivity information of the objective with respect to the virtual parameters that additional layers, if inserted, would offer. We consider fully connected feedforward networks with selected activation functions as well as residual neural networks. In numerical experiments, the proposed sensitivity-based layer insertion technique exhibits improved training decay, compared to not inserting the layer. Furthermore, the computational effort is reduced in comparison to inserting the layer from the beginning. The code is available at https://github.com/LeonieKreis/layer_insertion_sensitivity_based.