The architecture, training objective, adaptation methods, and systems ideas behind modern language models.
Scores query–key compatibility, normalizes it, and mixes value vectors.
Freezes the base matrix and learns a low-rank update, reducing trainable parameters.
Language models learn by assigning high probability to the next observed token.
A reusable technical reading framework for moving from vocabulary to working understanding. Use it alongside the formulas and papers below.
The architecture, training objective, adaptation methods, and systems ideas behind modern language models. Start by naming the representation, objective, and source of evidence in any system you study. This habit separates a useful model explanation from a list of buzzwords.
Use the learning outcomes as checks for understanding: Explain self-attention, positional information, and autoregressive decoding; Choose prompting, retrieval, fine-tuning, or preference optimization; Estimate memory, latency, and quality trade-offs. For each claim, ask what assumptions make it true, what data it needs, and how it could fail.
Translate each equation into a small experiment before treating it as memorized knowledge. Scaled dot-product attention: Scores query–key compatibility, normalizes it, and mixes value vectors. LoRA adaptation: Freezes the base matrix and learns a low-rank update, reducing trainable parameters. Next-token likelihood: Language models learn by assigning high probability to the next observed token.
Read primary work with a repeatable lens: problem, method, data, measurement, limits, and what would change your mind. This section starts with “Attention Is All You Need”, “BERT: Pre-training of Deep Bidirectional Transformers”, “LoRA: Low-Rank Adaptation of Large Language Models”. Follow citations outward only after you can explain the central claim in your own words.
Implement a minimal example using Scaled dot-product attention; record inputs, outputs, and a failure case.
Define a baseline, a success metric, a stress test, and the evidence required before you trust an improvement.
Choose one paper from the library, reproduce a small claim, then write down the questions its evaluation leaves open.
Primary sources and peer-reviewed work to deepen this page.
A. Vaswani et al.
Introduces the Transformer: parallel self-attention replaces recurrence for sequence modeling.
J. Devlin et al.
Shows that bidirectional masked-language pretraining transfers effectively to many NLP tasks.
E. J. Hu et al.
Adapts large models with compact low-rank trainable matrices while preserving the base weights.