Complete course notes

Pagina 1

Politecnico di Milano Artificial Neural Networks And Deep Learning Notes a.a. 2019-2020 Authors Alessio Russo Introito Matteo Moreschini March 28, 2020 Contents 1 From Perceptrons to Feed Forward Neural Networks 3 1.1 A Note on Maximum Likelihood Estimation . . . . . . . . . . . . . . . . 3 1.2 Neural Networks for Regression . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Neural Networks for Classification . . . . . . . . . . . . . . . . . . . . . . 5 2 Image Classification 8 2.1 Local (Spatial) Transformations . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Problem Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Nearest Neighborhood Classifier . . . . . . . . . . . . . . . . . . . . . . . 10 2.4 Linear Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4.1 Geometric Interpretation of a Linear Classifier . . . . . . . . . . . 12 3 Training and Overfitting 13 3.1 Dealing with Overfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.1.1 Early Stopping: Limiting Overfitting by Cross-Validation . . . . . 14 3.1.2 Weights Regularization . . . . . . . . . . . . . . . . . . . . . . . . 15 3.1.3 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 Tips and Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.1 ReLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2.2 Weights Inizialization . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2.3 Momentum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2.4 Batch Normalization . . . . . . . . . . . . . . . . . . . . . . . . . 22 4 Convolutional Neural Networks 24 4.1 The Feature Extraction Perspective . . . . . . . . . . . . . . . . . . . . . 24 4.2 Convolution . . .…