The compact mathematical language behind optimization, uncertainty, geometry, and learning dynamics.
Move parameters opposite the loss gradient; η controls the step size.
Measures orientation rather than magnitude, making it useful for comparing embeddings.
Updates a prior belief with evidence and a normalizing probability.
A reusable technical reading framework for moving from vocabulary to working understanding. Use it alongside the formulas and papers below.
The compact mathematical language behind optimization, uncertainty, geometry, and learning dynamics. Start by naming the representation, objective, and source of evidence in any system you study. This habit separates a useful model explanation from a list of buzzwords.
Use the learning outcomes as checks for understanding: Read vectors, matrices, distributions, and gradients fluently; Connect an objective function to its optimization update; Diagnose conditioning, variance, and numerical stability. For each claim, ask what assumptions make it true, what data it needs, and how it could fail.
Translate each equation into a small experiment before treating it as memorized knowledge. Gradient descent: Move parameters opposite the loss gradient; η controls the step size. Cosine similarity: Measures orientation rather than magnitude, making it useful for comparing embeddings. Bayes' rule: Updates a prior belief with evidence and a normalizing probability.
Read primary work with a repeatable lens: problem, method, data, measurement, limits, and what would change your mind. This section starts with “A Survey of Optimization Methods from a Machine Learning Perspective”, “Adam: A Method for Stochastic Optimization”, “Batch Normalization: Accelerating Deep Network Training”. Follow citations outward only after you can explain the central claim in your own words.
Implement a minimal example using Gradient descent; record inputs, outputs, and a failure case.
Define a baseline, a success metric, a stress test, and the evidence required before you trust an improvement.
Choose one paper from the library, reproduce a small claim, then write down the questions its evaluation leaves open.
Primary sources and peer-reviewed work to deepen this page.
S. Sun et al.
Organizes first-order, second-order, constrained, and adaptive optimization methods used in learning systems.
D. P. Kingma · J. Ba
Introduces an adaptive optimizer using running estimates of first and second moments of gradients.
S. Ioffe · C. Szegedy
Normalizes intermediate activations to make deep optimization more stable and faster.