Module 4 - Optimization for deep leaning

Table of Contents

Optimization for deep leaning

0:00 Recap
0:31 Plan
1:14 Optimization in deep learning
3:44 Gradient descent variants
7:58 Setting for the jupyter notebook
9:49 Vanilla gradient descent
12:14 Momentum
15:38 Nesterov accelerated gradient descent
18:00 Adagrad
20:06 RMSProp
22:11 Adam
24:39 AMSGrad
27:09 Pytorch optimizers

Slides and Practicals

• slides

• notebook in colab Code your optimizers.

References

• An overview of gradient descent optimization algorithms by Sebastian Ruder

• Gradient-based optimization A short introduction to optimization in Deep Learning, by Christian S. Perone

Edit this page on Last modified: May 28, 2025. Website built with Franklin.jl and the Julia programming language.