Skip to Main content Skip to Navigation
Conference papers

Unsupervised post-tuning of deep neural networks

Christophe Cerisara 1 Paul Caillon 1 Guillaume Le Berre 1
1 SYNALP - Natural Language Processing : representations, inference and semantics
LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : We propose in this work a new unsupervised training procedure that is most effective when it is applied after supervised training and fine-tuning of deep neural network classifiers. While standard regularization techniques combat overfitting by means that are unrelated to the target classification loss, such as by minimizing the L2 norm or by adding noise either in the data, model or process, the proposed unsupervised training loss reduces overfitting by optimizing the true classifier risk. The proposed approach is evaluated on several tasks of increasing difficulty and varying conditions: unsupervised training, posttuning and anomaly detection. It is also tested both on simple neural networks, such as small multi-layer perceptron, and complex Natural Language Processing models, e.g., pretrained BERT embeddings. Experimental results confirm the theory and show that the proposed approach gives the best results in posttuning conditions, i.e., when applied after supervised training and fine-tuning.
Document type :
Conference papers
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-02022062
Contributor : Christophe Cerisara <>
Submitted on : Thursday, April 15, 2021 - 3:17:25 PM
Last modification on : Sunday, April 18, 2021 - 3:22:14 AM

File

ijcnn.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02022062, version 2

Citation

Christophe Cerisara, Paul Caillon, Guillaume Le Berre. Unsupervised post-tuning of deep neural networks. IJCNN, Jul 2021, Virtual Event, United States. ⟨hal-02022062v2⟩

Share

Metrics

Record views

29

Files downloads

15