Gradient descent, how neural networks learn | Deep Learning Chapter 2
Core: Neural networks learn by minimizing a 'cost function' using gradient descent—an iterative optimization algorithm that calculates how to adjust thousands of weights and biases to reduce the network's error on training data.
One-line summary
This video explains the fundamental calculus problem behind neural network learning: how gradient descent uses the derivative of a cost function to iteratively adjust a network's parameters toward better performance.
Why this matters
As provocative as it is to describe a machine as learning, once you see how it works, it feels a lot less like some crazy sci-fi premise, and a lot more like a calculus exercise.
Neural network "learning" is fundamentally an optimization problem from calculus. Understanding gradient descent reveals that machine learning isn't magical—it's a systematic process of measuring error and making small, calculated adjustments to improve performance. This foundational concept applies to virtually all modern deep learning systems, from image recognition to language models.
Core concepts
Neural Network Structure
A neural network is a computational model structured in layers: an input layer, one or more hidden layers, and an output layer. For handwritten digit recognition, the example network processes a 28×28 pixel image (784 pixels) through the input layer. Each pixel's grayscale value (between 0 and 1) becomes the activation of an input neuron.
Each neuron in subsequent layers calculates its activation as a weighted sum of all previous layer activations, plus a bias term, transformed by an activation function like sigmoid or ReLU. For the example digit recognizer at 00:50, this simple architecture has 13,000 adjustable weights and biases that determine its behavior.
In plain terms, a neural network is a function: you feed in pixel values, and it spits out probabilities for each possible digit (0-9). The magic lies in finding the right 13,000 parameters so this function gives useful answers.
The Cost Function
The cost function quantifies how poorly the network performs. When you show the network an image of "3" but get a messy output with uncertain activations across all ten output neurons (as shown at 01:15), you need a mathematical way to measure that failure.
The cost function sums the squared differences between each output neuron's actual activation and its desired activation across all training examples. At 02:18, the formula is visualized:
Cost = (1/n) · Σᵢ Σⱼ (aⱼ⁽ⁱ⁾ − yⱼ⁽ⁱ⁾)²
where aⱼ is the actual activation and yⱼ is the desired activation (0 or 1) for output neuron j.
A smaller cost sum indicates better performance—when the network correctly lights up the "3" neuron and dims the others, the squared differences become tiny.
Think of it as the network's "report card": a single number that tells you how badly it's doing across thousands of test cases. The goal of learning is to find parameter values that minimize this number.
Gradient Descent: The 1D Case
To understand minimization, start simple. For a function with one input at 03:01, finding the minimum means finding where the slope (derivative) equals zero. But computing that analytically is often impossible for complex functions.
Instead, gradient descent works iteratively: start at any point, calculate the slope, and take a small step in the opposite direction (downhill). Repeat until you reach a valley (local minimum). The step size matters—too large and you overshoot; too small and convergence is painfully slow.
The analogy: a ball rolling down a hill naturally follows the slope to settle in a valley.
Gradient Descent: Multiple Dimensions
Real neural networks don't have one input—they have 13,000. The cost function becomes a landscape in 13,001-dimensional space (13,000 weights/biases as inputs, one cost value as output), impossible to visualize directly.
At 04:08, the video illustrates with just two inputs, creating a 3D surface with hills and valleys. The gradient is a vector pointing in the direction of steepest ascent—the direction that increases the function fastest. Its length indicates how steep that direction is.
The negative gradient points downhill—the direction of steepest descent. In the 13,000-dimensional case at 05:14, this negative gradient is a vector with 13,000 components, each telling you how to nudge a specific weight or bias to decrease cost most rapidly.
| Dimension | Gradient Component Meaning |
|---|---|
| 1D function | Single number (slope) indicating increase direction |
| 2D function | 2-element vector pointing uphill in xy-plane |
| 13,000D cost function | 13,000-element vector; each component indicates optimal nudge to one weight/bias |
The relative magnitudes of gradient components reveal which parameter changes have the greatest impact on reducing error.
Backpropagation
Computing the gradient of a cost function with 13,000 inputs by brute force (measuring how each weight affects cost independently) would be computationally prohibitive. Backpropagation is the efficient algorithm that solves this problem by propagating error backwards through the network.
As mentioned at 05:54, backpropagation is "the heart of how a neural network learns," though its mathematical details are reserved for the next video in the series. What matters here: it makes gradient descent tractable for deep networks.
Why Neurons Use Continuous Activations
Biological neurons fire or don't fire (binary). But artificial neurons use continuously ranging activation values (0 to 1, or other ranges) for a critical mathematical reason at 05:54: this ensures the cost function is smooth and differentiable.
Gradient descent requires calculating derivatives. Binary activations would create discontinuous jumps in the cost function—flat plateaus interrupted by sudden cliffs—where gradients don't exist or provide no useful direction information. Continuous activations create smooth valleys that gradients can navigate.
How the concepts fit together
The neural network is a function f(w₁, w₂, …, w₁₃₀₀₀) parameterized by its weights and biases. The cost function C(w₁, w₂, …, w₁₃₀₀₀) takes these same parameters as input and outputs a single number: the network's total error on training data.
Gradient descent uses the gradient ∇C to iteratively adjust parameters:
- Calculate
∇Cat the current parameter values (using backpropagation) - Update:
wᵢ ← wᵢ − α · (∂C/∂wᵢ)for alli - Repeat until cost stops decreasing significantly
The learning rate α controls step size. This process navigates the 13,000-dimensional landscape, following valleys downhill until finding a local minimum where the network performs well.
Correcting misconceptions
Misconception: Neural networks learn by discovering human-intuitive patterns like specific edges or loops in their hidden layers, similar to how human vision works.
Actually: While hierarchical pattern recognition (edges → loops → digits) is a motivating idea, simple networks often find effective but non-intuitive patterns. At 06:44, when visualizing the weights connecting to hidden neurons as pixel patterns, they appear almost random—loose patterns in the middle rather than clear edges. The network achieves 96-98% accuracy through complex, seemingly random weight configurations that don't correspond to human-interpretable features. It may essentially memorize training data rather than learn conceptual features.
Misconception: A network that performs well on classification understands its inputs generally and can express uncertainty with unfamiliar data.
Actually: Networks trained exclusively on perfectly structured data (centered digits in the MNIST dataset) become overconfident. At 06:30, when fed completely random noise, the example network confidently classifies it as a specific digit (e.g., "5") rather than expressing uncertainty. This happens because the cost function never incentivized uncertainty or taught the network to recognize out-of-distribution inputs. The network only knows to classify, not to say "I don't know."
Real-world limitations and modern insights
The simple network architecture discussed achieves respectable 96-98% accuracy on handwritten digits, but at 07:54, two key limitations emerge:
- Lack of interpretable features: Modern deep learning reveals that networks don't always learn human-intuitive representations
- Overconfidence on out-of-distribution data: Without proper training incentives, networks fail to express appropriate uncertainty
Training exclusively on highly constrained, perfectly structured data creates networks that lack robustness to real-world variation and cannot recognize when inputs fall outside their training distribution.
At 09:20, researcher Leisha Lee discusses modern findings on network learning behavior:
| Training Condition | Learning Behavior |
|---|---|
| Properly labeled data | Gradient descent finds structure quickly; loss drops rapidly |
| Randomly shuffled labels | Gradient descent still finds solutions, but more slowly (linearly); the network memorizes rather than learns patterns |
This reveals that gradient descent can "solve" the optimization problem even when no meaningful patterns exist—the network simply memorizes the training set. The quality of local minima found by gradient descent is surprisingly good, even in high-dimensional spaces where intuition suggests many poor solutions might exist.
Going deeper
To deepen your understanding of neural network learning:
- Backpropagation algorithm: Study the mathematical details of how gradients are efficiently computed through chain rule applications (covered in the next video of this series)
- Optimization algorithms: Explore variants like SGD, Adam, RMSprop that improve on basic gradient descent
- Regularization techniques: Learn methods (dropout, L2 regularization) that prevent memorization and encourage generalization
- Loss functions: Investigate alternatives to mean squared error (cross-entropy, hinge loss) and their properties
- Activation functions: Understand why ReLU often outperforms sigmoid in deep networks
- Learning rate scheduling: Study techniques for adapting step size during training
- Michael Nielsen's free online book "Neural Networks and Deep Learning" (recommended at 08:26)
Practice by implementing gradient descent on simple functions before tackling neural networks. Visualize 2D cost landscapes to build intuition before working in high dimensions.
Related concepts
- Deep learning
- Supervised learning
- MNIST dataset
- Multivariable calculus
- Optimization theory
- Overfitting vs generalization
- Stochastic gradient descent
- Neural network architectures
- Convolutional neural networks
Key timestamps
- 00:00 — Introduction & Video Goals: Recap & Gradient Descent
- 00:50 — Neural Network Structure & Goal: Handwritten Digit Recognition
- 02:18 — Introducing the Cost Function: Quantifying Network Lousiness
- 03:01 — Minimizing the Cost Function: The Simple 1D Case
- 04:08 — Gradient and Gradient Descent: The 2D Case
- 05:14 — Gradient Descent in High Dimensions (13,000 inputs)
- 05:54 — Backpropagation & Why Smoothness Matters
- 06:44 — Performance & Unintuitive Learning of the Example Network
- 07:54 — Why the Simple Network Doesn't Learn Human-like Features
- 09:20 — Interview Snippet: Modern Deep Learning & Memorization vs. Structure