Note on Backpropagation in Neural Networks

Xin Jin (1)

1 UPMC - Université Pierre et Marie Curie - Paris 6

This note intends to facilitate low level implementation by providing an analytical perspective on neural networks. Different feedforward and recurrent neural networks are dissected through a derivation of the backpropagation update. We choose Multilayer Perceptron (MLP) which possesses the basic architecture of deep artificial neural network as a departure of introduction. Sigmoid Cross-Entropy loss is applied to MLP for an exemplification of multi-label classification. We then turn to introduce Convolutional Neural Network (CNN)-an intricate architecture which adopts filter and sub-sampling to realize a form of regularization. In the end, we illustrate Backpropagation Through Time (BPTT) to elicit Exploding / Vanishing Gradients problem and Long short-term memory (LSTM).

Domaines: Intelligence artificielle [cs.AI]

Xin Jin. Note on Backpropagation in Neural Networks. 2019. ⟨hal-02265247⟩