Tutorial notebooks

Currently Playing: Tutorial 4: Optimization and Initialization (Part 2)

In this tutorial, we will review techniques for optimization and initialization of neural networks. When increasing the depth of neural networks, there are various challenges we face. Most importantly, we need to have a stable gradient flow through the network, as otherwise, we might encounter vanishing or exploding gradients. This is why we will take a closer look at the following concepts: initialization and optimization. This notebook is part of a lecture series on Deep Learning at the University of Amsterdam. The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io. Link to the notebook: https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial4/Optimization_and_Initialization.html Correction: Note that in the shown Adam implementation, the bias correction factor for the momentum was missing in the update step. The notebook has been updated accordingly (see here: https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial4/Optimization_and_Initialization.html#Optimization). The implications and results for the Adam optimizer stay the same (only minor differences). 00:00 Setup 01:16 Optimizer template 02:59 SGD 03:40 SGD with Momentum 04:45 Adam 07:10 FashionMNIST comparison 08:48 Pathological curvatures 11:54 Steep optima 14:10 What optimizer to take 17:10 Conclusion