How to understand gradient descent for an interview
Understand gradient descent as a causal optimization process, including learning rate, convergence, divergence, and the questions interviewers use to test it.
The short version
Gradient descent minimizes a differentiable objective by repeatedly measuring the local slope with respect to the parameters and moving those parameters in the opposite direction. The gradient supplies direction and relative sensitivity; the learning rate controls step size. A useful explanation must also cover why steps can oscillate, diverge, stall, or settle somewhere that is not the best possible solution.
The problem it solves
Training often means choosing many parameter values that make a loss small. Solving for the exact minimum may be unavailable or too expensive, so gradient descent uses local information to improve the parameters iteratively. The gradient is not the destination; it is the best local description of which parameter changes increase the objective fastest. Negating it produces the steepest local decrease under the usual Euclidean geometry.
The smallest working case
For a one-parameter quadratic loss, the derivative tells you whether the current parameter sits left or right of the minimum and how sharply the loss changes there. Subtracting a fraction of that derivative moves toward the bottom. As the parameter approaches the minimum, the gradient shrinks and so do the updates.
This simple case explains the update rule, but it hides the difficulties introduced by many dimensions, noisy mini-batches, uneven curvature, saddle points, and non-convex objectives.
Why the learning rate matters
A very small learning rate can make useful progress painfully slow. A large rate can cross the valley repeatedly, oscillate, or increase loss until the computation diverges. The right value depends on the scale and geometry of the objective, which is why schedules and adaptive optimizers exist.
When diagnosing training, read the loss curve as evidence. Smooth but slow decline, repeated oscillation, explosive growth, and long plateaus point to different interactions among step size, data, initialization, and curvature.
What a strong interview answer includes
- The objective being minimized.
- The gradient as parameter sensitivity, not merely “the slope.”
- The negative-gradient update and the role of learning rate.
- The distinction between full-batch, stochastic, and mini-batch estimates.
- At least one failure mode and a diagnostic signal.
Common weak answers
- Saying the gradient points toward the minimum—it points toward local increase.
- Claiming gradient descent always finds the global minimum.
- Treating the learning rate as a speed control with no effect on stability.
- Naming Adam or momentum without explaining the problem each modification addresses.
Could you answer this cold?
Training loss falls for several steps, then begins oscillating and eventually explodes. Explain the most likely optimization mechanism, what you would inspect, and why your first intervention should help.
A strong answer should
- Connects overshooting to learning rate and local curvature.
- Uses the loss curve or gradient norms as evidence.
- Distinguishes numerical problems from ordinary overfitting.
- Explains the intervention causally rather than merely naming a lower learning rate.
No account required for the guided preview. Your answer is not placed in the URL.
Primary sources
This guide is an original learning and interview-preparation synthesis. AI assisted with editing; the structure, claims, and cold-answer rubric were reviewed by MyPage.