Articles
Posts Tags Categories Random
Partners
About
About Author Portfolio Terms of Use Privacy Policy Time Travel Donate
More
Settings Funny Tools Useful Websites Student Works Message Board Recent Comments

Contents

From Gradient Descent to Stochastic Gradient Descent

 Included in Machine Learning
 Published   Updated    1519 words   8 minutes   views
https://raw.githubusercontent.com/Josh-test-lab/website-assets-repository/refs/heads/main/posts/From%20Gradient%20Descent%20to%20Stochastic%20Gradient%20Descent/cover%20image.webp

Cover image was generated by ChatGPT.

Gradient descent is an iterative method used in most modern models to help find optimal parameters. However, in today’s era of big data, the massive amount of data results in significant time costs for gradient descent. One key goal is to compute parameters quickly without sacrificing accuracy.

What is a Gradient

A gradient is a generalization of the derivative for multivariable functions, describing the rate of change of a function along each variable’s direction. The gradient vector points in the direction of the steepest increase of the function at a point, and its magnitude represents the rate of change in that direction. Simply put, if the derivative in one dimension represents the slope at a point on a curve, the gradient is the vectorized expression of the “slope” in higher-dimensional space.

Example

Take the example from Wikipedia. Consider a mountain, where the height at coordinate point $(x, y)$ is given by the function $H(x,y)$. The gradient $\nabla H(x,y)$ describes the direction of the steepest slope at that point, and its magnitude reflects how steep the slope is.

The figure below shows the function $H(x, y) = x^2 + y^2$ at point $(1, 1)$. The red vector represents the gradient direction at that point; the blue vector is a particular directional vector, and the light blue vector is the projection of that direction onto the gradient.

https://raw.githubusercontent.com/Josh-test-lab/website-assets-repository/refs/heads/main/posts/From%20Gradient%20Descent%20to%20Stochastic%20Gradient%20Descent/gradient_example.webp
Example image.

For a scalar function $f: \mathbb{R}^n \to \mathbb{R}$, the gradient is expressed as $\nabla f: \mathbb{R}^n \to \mathbb{R}^n$, where $\nabla$ is the vector differential operator:

$$ \nabla f(\boldsymbol{x}) = \begin{pmatrix} \frac{\partial f}{\partial x_1}(\boldsymbol{x}) \\ \vdots \\ \frac{\partial f}{\partial x_n}(\boldsymbol{x}) \end{pmatrix} $$

where $\boldsymbol{x} = (x_1, \cdots, x_n)$. Furthermore:

$$ \nabla f(\boldsymbol{x}) \cdot \boldsymbol{v} = \frac{\partial f}{\partial \boldsymbol{v}}(\boldsymbol{x}) = df_{\boldsymbol{x}}(\boldsymbol{v}) $$

Here, $\boldsymbol{v}$ is any unit vector, and $df_{\boldsymbol{x}}(\boldsymbol{v})$ is the rate of change of $f$ at point $\boldsymbol{x}$ along direction $\boldsymbol{v}$.

We aim to use gradient descent to minimize a loss function, finding the set of parameters that best describe the training data and approximate the optimal solution of the model.

Gradient Descent

Gradient descent (GD) is one of the fundamental optimization algorithms in machine learning and deep learning, used to find the minimum of a differentiable loss function. It works by computing the loss function with the current model parameters (weights and biases) and updating the parameters along the negative gradient direction. In practice, “gradient descent” usually refers to Batch Gradient Descent (BGD), which computes the gradient of the loss function using the entire training dataset.

$$ w_{n + 1} = w_n - \eta \nabla L (w_n) $$

Here, $w_n$ is the parameter vector at the $n$-th iteration; $w_{n+1}$ is the updated parameter; $\eta > 0$ is the learning rate controlling the step size; $L(w)$ is the loss function measuring model performance on the training data; $\nabla L(w_n)$ is the gradient at the current parameters $w_n$. By iterating repeatedly, we gradually approach the minimum of the loss function.

If using mean square error (MSE) as the loss function to calculate and update weight parameters, suppose $(x_1, \cdots, x_m)$ are input sample vectors, where $x_i \in \mathbb{R}^n$ represents the $i$-th sample’s feature vector; $(y_1, \cdots, y_m)$ are output labels, where $y_i \in \mathbb{R}$ is the true value of the $i$-th sample; the weight vector is $\boldsymbol{w} \in \mathbb{R}^n$. Ignoring bias $\varepsilon_i$, the loss function is:

$$ L(\boldsymbol{w})=\frac{1}{2m}\sum_{i=1}^m (y_i - x_i^\top \boldsymbol{w})^2 $$

The factor $2m$ simplifies gradient computation. The gradient is:

$$ \begin{align*} \nabla L(\boldsymbol{w}) & = \frac{1}{2m}\sum_{i=1}^m 2 (x_i^\top \boldsymbol{w} - y_i) x_i \\ & = \frac{1}{m}\sum_{i=1}^m (x_i^\top \boldsymbol{w} - y_i) x_i \end{align*} $$

As model size and data volume grow rapidly, BGD, which requires computing the gradient with all parameters, can become slow and resource-intensive. Therefore, stochastic gradient descent and its variants are widely used in practice to improve efficiency and convergence speed.

https://raw.githubusercontent.com/Josh-test-lab/website-assets-repository/refs/heads/main/posts/From%20Gradient%20Descent%20to%20Stochastic%20Gradient%20Descent/gradient%20descent%20cost%20and%20weight.webp
Relationship between cost and weights in gradient descent, captured from Analytics Vidhya at August 30, 2025, the author is Crypto1.

Stochastic Gradient Descent

Unlike BGD, which updates parameters after computing the gradient for the entire dataset, stochastic gradient descent (SGD) updates parameters using a randomly selected single sample per iteration. In the $n$-th iteration, a random index $i_n \in \{ 1, \cdots, m \}$ is selected from the training set, and only the gradient of sample $(x_{i_n}, y_{i_n})$, $\nabla L_{i_n} (w_n)$, is used to update the parameters:

$$ w_{n + 1} = w_n - \eta \nabla L_{i_n} (w_n) $$

This means the model parameters are updated after processing each sample. Compared to BGD, SGD updates more frequently, but each gradient estimate contains high stochastic noise. This randomness sometimes helps SGD escape shallow local minima or saddle points, though it also causes a non-monotonic convergence path with oscillations near minima. With sufficient iterations and proper learning rate decay, SGD can converge in expectation to a global minimum or a good local minimum.

The main advantage of SGD is computational and memory efficiency, as only a single sample is read and computed at a time, making large-scale training feasible. Its random sampling also helps avoid poor local minima. However, since each gradient estimate is based on a single sample, fluctuations can be large, leading to oscillations in the loss curve. Too high a learning rate can even cause divergence. Hence, strategies like gradually decaying learning rates are commonly used to stabilize convergence.

Mini-batch Gradient Descent

Mini-batch gradient descent (MBGD) is a method between BGD and SGD. In each iteration, the training set is randomly shuffled and divided into small batches. Gradients are computed and parameters updated using multiple samples from each batch.

Compared to BGD, MBGD updates using fewer data points, reducing computational cost. Compared to SGD, MBGD reduces gradient noise by averaging over multiple samples, resulting in more stable updates. This balance, more stable than SGD and more frequent than BGD, makes MBGD the mainstream approach in deep learning training.

Mathematically, if a mini-batch $B_n \subset \{ 1, 2, \dots, m \}$ of size $b$ is randomly selected in the $n$-th iteration:

$$ \nabla L_{B_n}(w_n) = \frac{1}{b} \sum_{i \in B_n} (x_i^\top w_n - y_i) x_i $$

The corresponding parameter update rule is:

$$ w_{n+1} = w_n - \eta \nabla L_{B_n}(w_n) $$

Here, $b$ is the batch size (common values: $32, 64, 128$); $B_n$ is the index set of the batch in the $n$-th iteration. When $b = 1$, MBGD reduces to SGD; when $b = m$, it reduces to BGD.

MBGD benefits from parallel computation, allowing efficient GPU execution for each batch, greatly improving update efficiency. Compared to SGD, MBGD updates are smoother, reducing oscillations caused by gradient fluctuations. The partial sample randomness also helps the model escape poor local minima, improving overall convergence.

However, MBGD requires careful selection of batch size. Too small a batch increases gradient noise, destabilizing updates. Too large a batch approaches BGD, losing stochastic and efficiency advantages. In practice, batch size is tuned according to data scale, hardware, and model characteristics to achieve optimal training.

Comparison of the Three Methods

MethodData per IterationUpdate FrequencyAdvantagesDisadvantagesSuitable Scenario
Gradient Descent (BGD)All samples
$m$		LowMost precise update direction, smooth convergenceHigh computational cost, high memory demandSmall datasets, simpler models
Stochastic Gradient Descent (SGD)Single sample
$1$		HighFast computation, can escape local minimaHigh noise, noticeable oscillations, requires learning rate decayLarge datasets
Mini-batch Gradient Descent (MBGD)Mini-batch
$b$		MediumEfficient computation, parallelizable, smoother gradientsBatch size needs tuning. Too large → BGD, too small → SGDMainstream deep learning training

Using $f(x) = (x - 3)^2$ as the loss function with randomly generated data, MBGD uses a batch size of 5, 50 iterations, and learning rate of 0.1.

https://raw.githubusercontent.com/Josh-test-lab/website-assets-repository/refs/heads/main/posts/From%20Gradient%20Descent%20to%20Stochastic%20Gradient%20Descent/methods_comparison.webp
Comparison of the three gradient descent methods.

From the figure, BGD has the most precise update direction and smooth convergence; SGD updates with only one sample, moving quickly but oscillating; MBGD combines the advantages of both, maintaining efficiency while achieving relatively stable convergence.

Conclusion

In practice, MBGD is the standard approach for deep learning training. However, as data and model complexity increase, relying solely on gradient descent is insufficient. Adaptive learning rate algorithms (e.g., Adam, RMSProp, Adagrad) were developed based on MBGD to further improve convergence speed and stability. Understanding the differences, advantages, and disadvantages of these fundamental methods helps select appropriate optimization strategies and better grasp the mathematical principles behind deep learning training.

Further Learning

References

Updated on 2025-08-31 14:30 UTC+0800   c334c4b9
Creative Commons License
 Machine Learning, Algorithm, Model, Optimization, Gradient Descent, Deep Learning
Back | Home