Tutorial notebooks

Currently Playing: Tutorial 4: Optimization and Initialization (Part 1)

In this tutorial, we will review techniques for optimization and initialization of neural networks. When increasing the depth of neural networks, there are various challenges we face. Most importantly, we need to have a stable gradient flow through the network, as otherwise, we might encounter vanishing or exploding gradients. This is why we will take a closer look at the following concepts: initialization and optimization. This notebook is part of a lecture series on Deep Learning at the University of Amsterdam. The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io. Link to the notebook: https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial4/Optimization_and_Initialization.html 00:00 Introduction 02:16 Preparation 07:24 Initialization (intro) 08:12 Constant initialization 10:12 Constant variance 13:01 Variance preserving inits 17:00 Xavier initialization 22:35 Kaiming initialization