λ
home posts study uses projects

Gradient Descent

Jul 18, 2024

machine-learning

Gradient Descent is the most popular Optimizer for machine learning models.

How it works

It involves the following steps

  • Go over the entire dataset
  • Compute the gradients of the weights & biases, using Backward-Propagation with respect to the Cost Function
  • Average the gradients over the entire dataset
  • Add the gradients to the weights & biases while also multiplying them by a Learning Rate

Batching

You can introduce some form of batching to the process

  • Mini-batch Gradient Descent allows you to adjust the parameters every N batches instead of after processing the entire dataset. This is less computationally expensive.
  • Stochastic Gradient Descent is like Mini-batch Gradient Descent, but with N = 1. This means that you tweak the parameters after every sample in the data set.

LλBS

All systems up and running

Commit c0daef5, deployed on Oct 01, 2024.