Four points, two classes, one line that does not exist. XOR is the smallest task a linear model cannot do and a hidden layer can — not because the hidden layer is deeper, but because it changes the coordinates in which the output layer works.
00 / The hookXOR, features, and the object that matters.
Take the four points (0,0), (0,1), (1,0), (1,1). Label the first and last 0, the middle two 1. This is exclusive or: the output is 1 when the inputs differ. A linear classifier computes w1x1 + w2x2 + b and predicts 1 when the score is positive. Can any w1, w2, b fit the labels?
No, and the proof is four inequalities. The point (0,0) is negative, so b < 0. The point (1,1) is negative, so w1 + w2 + b < 0. The positives require w2 > −b and w1 > −b, hence w1 + w2 + b > −b > 0, contradicting the requirement. There is no line. This is the argument Minsky and Papert made central to Perceptrons (1969), and it is correct: a single-layer neural network cannot represent XOR.
The fix is to learn a new representation before the linear readout. Let h1 = ReLU(x1 − x2) and h2 = ReLU(x2 − x1), and predict 1 when h1 + h2 > 0.5.
| Input x | Label | h1 | h2 | h1 + h2 | Prediction |
|---|---|---|---|---|---|
| (0, 0) | 0 | 0 | 0 | 0 | 0 |
| (0, 1) | 1 | 0 | 1 | 1 | 1 |
| (1, 0) | 1 | 1 | 0 | 1 | 1 |
| (1, 1) | 0 | 0 | 0 | 0 | 0 |
The map is not a rotation. A linear map preserves linear separability, so no linear change of coordinates could separate XOR. The hidden layer is piecewise linear: it folds the unit square along x1 = x2. The two negative corners lie on the fold and are identified at the origin; the positives are pushed onto the axes. One line then separates the classes.
Look at what the representation threw away. The map depends only on d = x1 − x2: if d > 0 the state is (d, 0), if d < 0 it is (0, −d), and if d = 0 it is (0, 0). The sum x1 + x2 is gone. For XOR that is the nuisance direction, so disposing of it is free. For a task that asks “is the count at least 2?”, the same representation would be blind. A representation is a choice of what to ignore, and the choice is only good relative to a task.
A representation is a map φ: X → Rd. A task is linearly separable in φ if there is a w and b with y = sign(w·φ(x) + b) on the data. A deep network learns φ and the readout at once. The readout is almost always linear; the interesting object is φ.
The two-unit construction is a proof of concept, not a training result. A network with two hidden units can represent XOR; whether gradient descent finds a solution depends on initialization, step size, and the loss. The lab lets you watch both regimes on noisy 2D data.
01 / Learning as optimizationEmpirical risk, gradients, and the chain rule at scale.
A supervised model is a function fθ with parameters θ, a data distribution P, and a per-example loss function ℓ(fθ(x), y). We do not have P, so we minimize the risk on a sample:
The population risk is R(θ) = E(x,y)~P[ℓ(fθ(x), y)]; the difference is the generalization gap. For classification, use cross-entropy: softmax over K classes, ℓ = −log py, computed with log-sum-exp.
The gradient is the direction of steepest increase, so gradient descent steps against it:
The step size is not cosmetic: if R̂ is L-smooth, any 0 < η < 2/L decreases the loss. Too large bounces; too small crawls.
Backpropagation is the chain rule, scheduled
A network is a composition fθ = fL ∘ … ∘ f1. The derivative of the loss with respect to an early parameter is a product of Jacobians. Backpropagation evaluates those products from the output backwards: each node multiplies the incoming gradient by its local Jacobian and passes it on. Its cost is a small constant multiple of the forward pass, independent of the parameter count. Activations are cached because each local derivative is evaluated at the forward value: the derivative of tanh is 1 − tanh².
A worked step, by hand
Take x = 1, y = 2, and z = wx + b, a = ReLU(z), ŷ = va + c, with L = ½(ŷ − y)². Start at w = 0.5, b = 0, v = 1, c = 0, η = 0.1.
Forward: z = 0.5, a = 0.5, ŷ = 0.5, L = 1.125. Backward: ∂L/∂ŷ = −1.5; then ∂L/∂v = −0.75, ∂L/∂c = −1.5, ∂L/∂a = −1.5. Since z > 0, ∂L/∂z = −1.5, so ∂L/∂w = −1.5 and ∂L/∂b = −1.5. Subtract η times each gradient.
| Quantity | Before | Gradient | After update |
|---|---|---|---|
| w | 0.500 | −1.500 | 0.650 |
| b | 0.000 | −1.500 | 0.150 |
| v | 1.000 | −0.750 | 1.075 |
| c | 0.000 | −1.500 | 0.150 |
| z | 0.500 | −1.500 | 0.800 |
| a | 0.500 | −1.500 | 0.800 |
| ŷ | 0.500 | −1.500 | 1.010 |
| L | 1.125 | — | 0.490 |
The loss fell from 1.125 to 0.490. Try η = 1: the parameters become w = 2, b = 1.5, v = 1.75, c = 1.5, the output is 7.625, and the loss is about 15.8. The gradient pointed downhill; the step was too long. This is why schedules exist.
Minibatches and adaptive steps
Stochastic gradient descent uses a minibatch Bt instead of the full data:
For a uniform batch, E[gt] = ∇R̂ and Var(gt) ≈ σ²/B. The estimate is unbiased but noisy. Batch size changes both the number of updates and the character of the solution; the folklore that small batches generalize better is a tendency, not a theorem.
Momentum averages past gradients and helps across plateaus. Adam (Kingma & Ba, 2015) adds a per-parameter second moment:
with bias corrections m̂t, v̂t; the defaults are β1 = 0.9, β2 = 0.999, ε = 10−8.
Minimizing R̂ optimizes a proxy. If the proxy rewards the wrong cue — a watermark, a hospital label, a background texture — the model learns it perfectly. A low loss is evidence about the sample and the objective, not about the world. Section 4 names this failure: shortcut learning.
02 / Architectures as inductive biasesA layer is an assumption about what should be easy.
An inductive bias is the set of assumptions a learner uses to generalize beyond its data. Without one, no finite sample determines a function on unseen inputs: infinitely many functions fit the sample and disagree everywhere else. Architecture is the most important bias in deep learning because it decides which functions are easy to represent and which are effectively unreachable with the data available.
An MLP composes affine maps and nonlinearities: f(x) = WLφ(…φ(W1x + b1)…). Its bias is weak: every input can interact with every other, and nothing is invariant. Universal approximation (Hornik, Stinchcombe & White, 1989) says one hidden layer with enough units can approximate any continuous function on a compact set. It is an existence statement: it says nothing about finding the units from n examples. Some functions also need exponentially many shallow units but only polynomially many deep ones (Telgarsky, COLT 2016).
A convolutional network replaces the dense matrix with a small kernel applied at every position. In library notation the operation is cross-correlation:
Two assumptions are built in: locality, so a feature depends on a small neighbourhood, and weight sharing, so one detector is useful everywhere. The consequence is translation equivariance, conv(Tδx) = Tδconv(x); pooling adds approximate invariance. A dense layer from 32×32×3 to 64 units has 196,672 parameters; a 3×3 convolution from 3 to 64 has 1,792, about 110 times fewer. The bias is wrong when position is meaningful.
A recurrent network shares weights across time and carries a state:
The bias is order and variable length. The weakness is the gradient: the derivative of ht with respect to ht−k is a product of k Jacobians, which vanishes or explodes with their singular values. The LSTM (Hochreiter & Schmidhuber, 1997) mitigates this with gated paths; recurrence is also hard to parallelize.
Self-attention computes a soft lookup. Each position forms a query, key, and value; the query scores all keys, the scores are normalized, and the values are mixed:
A Transformer stacks multi-head attention with feed-forward blocks, residuals, and layer normalization. Without positional encodings it is permutation equivariant; positions are added because language is ordered. The bias is weak: locality and syntax must be learned from data, a strength at scale and a liability on small data. Full attention costs O(n²).
A graph network passes messages along edges: mv = Σu∈N(v) M(hu, hv, euv), then h′v = U(hv, mv). It is permutation equivariant by construction and bounded by 1-WL (Xu et al., 2019); Lecture 04 takes this apart.
| Architecture | Built-in assumption | What it saves | Where it fails |
|---|---|---|---|
| MLP | Only smoothness; all inputs may interact | Generality | Needs data to learn locality |
| CNN | Locality and translation equivariance | Parameters and samples | Position matters; rotation and scale |
| RNN | Order and weight sharing over time | Variable length | Long-range gradients; sequential compute |
| Transformer | Permutation equivariance plus positions | Long-range mixing; transfer | Data hunger; quadratic cost |
Four architectures, one question
MLP
Every input can reach every unit; the prior is only smoothness.
CNN
Locality and weight sharing: a detector learned in one place works everywhere.
RNN
A state carried through time with shared weights; the prior is order.
Transformer
Every position can read every other; the prior is weak until positions are added.
A short lineage
McCulloch & Pitts
Minsky & Papert, Perceptrons
Backpropagation
AlexNet
Transformer
03 / Representation learningWhat the objective rewards, the representation keeps.
Representation learning is learning the features themselves rather than hand-engineering them. Bengio, Courville and Vincent define it as learning representations that make it easier to extract useful information when building classifiers or other predictors (IEEE TPAMI, 2013). The definition is downstream-relative: a representation is good for a task, not good in the abstract.
- Sufficiency. The label is still recoverable: I(y; φ(x)) close to I(y; x).
- Invariance. Nuisance factors do not change the vector.
- Smoothness. Nearby inputs map to nearby vectors.
- Linear accessibility. A linear probe on top solves the task, the usual diagnostic for frozen features.
No objective delivers all four. A reconstruction objective keeps the noise because the noise is part of the pixels; a label objective keeps what the label needs and discards the rest. The discarded information is not a bug; it is the network doing what it was told.
An embedding is a learned vector for a discrete object. Mikolov et al. (2013) reported that king − man + woman is close to queen in a skip-gram space: a reported regularity, not a law of semantics. A prediction task forces useful distinctions into the coordinates, and some become linearly accessible.
Self-supervised learning manufactures supervision from the data itself. Masked language modelling hides tokens and predicts them; next-token prediction does the same at every position; contrastive vision models match two augmented views. The advantage is scale without labels. The cost is that the pretext task decides the representation: a model trained to predict captions learns what captions mention, not everything visible.
A contrastive objective pulls a positive pair together and pushes negatives apart. The standard form is InfoNCE:
The temperature τ controls the separation. The failure mode is collapse: without negatives, mapping every input to one vector minimizes the objective; later methods use a predictor and a stop-gradient (BYOL) or a redundancy term. CLIP applies the idea across modalities: an image encoder and a text encoder are trained so a caption and its image are close, and zero-shot classification compares an image to text embeddings of class names (Radford et al., 2021).
An autoencoder minimizes reconstruction error, ‖x − D(E(x))‖², with the bottleneck as the representation. A linear autoencoder with squared error spans the same subspace as PCA; a nonlinear one can learn a curved manifold. Reconstruction is a strange objective for semantics: a pixel-perfect autoencoder may spend capacity on texture and discard the label.
A variational autoencoder makes the bottleneck stochastic. With prior p(z), decoder pθ(x|z), and encoder qφ(z|x), training maximizes a lower bound:
The reparameterization trick, z = μ + σ ⊙ ε with ε ~ N(0, I), makes sampling differentiable. The price is a bound, not the likelihood; reconstructions are often blurry, and the decoder can learn to ignore z (posterior collapse). GANs instead play a minimax game, with mode collapse as the characteristic failure (Goodfellow et al., 2014).
| Objective | Preserves | Tends to discard | Characteristic failure |
|---|---|---|---|
| Reconstruction (AE) | What redraws the input | Semantic factors the pixels do not force | Capacity spent on nuisance |
| Variational bound (VAE) | A smooth latent neighbourhood | Sharp detail; sometimes the latent | Posterior collapse; blur |
| Contrastive (CLIP) | What positive pairs share | What augmentation or caption omits | Collapse; shortcut pairs |
| Next token | Predictable structure in text | What the corpus never makes predictable | Fluency without fact |
| Supervised | What the label needs | Everything else, including transfer | Shortcuts; poor transfer |
The table is the argument: the objective chooses what survives.
04 / GeneralizationWhy a low training loss is not a promise.
Generalization is not one thing. A model can be excellent on the training distribution, fragile under a small shift, blind to a novel combination, and unable to answer a counterfactual. The word covers four questions, and a benchmark answers one.
| Kind | What is held out | What failure looks like |
|---|---|---|
| iid | Fresh points from the same distribution | Train accuracy above test accuracy |
| Out-of-distribution | A different domain, sensor, hospital, language | Accuracy collapses; confidence often does not |
| Compositional | Novel combinations of familiar parts | High iid accuracy, chance on the new combination |
| Causal | Interventions that change the mechanism | Correlations survive; the right action does not |
For squared loss, expected error decomposes:
This is the bias–variance decomposition. The bias is the gap between the average prediction and the truth; the variance is the spread across training sets; σ² is noise no model can remove. Rigid models have high bias, flexible ones high variance. The classical prescription is the validation U-curve, which is right for a fixed family and procedure.
Regularization trades variance for bias: weight decay, dropout, early stopping, augmentation, label smoothing, weight sharing. Each is a prior, and none is free: a prior that is wrong for the data costs accuracy, and the cost is invisible on a validation set that shares the same bias.
The puzzle that broke the simple story
Zhang et al. (2017) trained standard image classifiers on CIFAR-10 with randomized labels. The networks fit the training set almost perfectly and had chance accuracy on the test set. The same architectures fit the real labels and generalize well. So capacity in the “can fit anything” sense does not predict generalization, and the classical uniform-convergence bounds are vacuous at this scale. Something about the data, the architecture, and the optimizer selects a good solution among the many that fit. What that something is remains open.
Double descent
Belkin et al. (2019) observed what is now called double descent: test error is not always a U-curve. As capacity grows, error falls, peaks at the interpolation threshold — just enough capacity to fit the training data exactly — and then falls again. Near the threshold the interpolating solution is sensitive to the data; far past it there are many interpolating solutions and the optimizer's implicit bias tends to pick a smoother one. That story is cleanest for linear models and random features; for deep networks it is a picture, not a derivation.
The curve depends on label noise, the optimizer, early stopping, and how capacity is defined. It can disappear with regularization, and a model can sit on the good side for the wrong reason. Read it as a warning against a single U-curve, not as a design recipe.
Distribution shift, shortcuts, and composition
Distribution shift is the default reason deployed models fail. Covariate shift: P(x) changes, P(y|x) fixed. Label shift: P(y) changes. Concept shift: P(y|x) changes. A model trained on one hospital, camera, or language is a model of that distribution as much as of the task.
Domain generalization trains on several domains and tests on an unseen one. Gulrajani and Lopez-Paz (2021) built DomainBed and found that, with equal tuning budgets, plain empirical risk minimization is competitive with the specialized algorithms. A method that beats a badly tuned baseline has not shown that it generalizes better.
Shortcut learning is a model exploiting a cue that correlates with the label in the training data but is not the intended mechanism (Geirhos et al., 2020). A classifier can learn the background instead of the object, the scanner instead of the pathology, the annotation style instead of the argument. The training loss is low because the cue is genuinely predictive on that sample. The failure appears when the cue is removed or reversed. More data from the same distribution may not help; the intervention has to change the data-generating process.
Compositional generalization is handling novel combinations of familiar parts. Lake and Baroni's SCAN (2018) maps commands to action sequences and includes a split with new combinations of the same primitives. Sequence-to-sequence models that fit the random split fail on the compositional split. The result is diagnostic, not decisive: later work with augmentation narrowed the gap, so SCAN is a probe of inductive bias, not a verdict on an architecture.
In the lab, the compositional split holds out one quadrant of a 2D XOR task. With the default model and seed, a one-hidden-layer network reaches 100% on the iid test set, 0% on the held-out quadrant, and stays near 100% confident there. A linear model fails the same way for a simpler reason: the three seen quadrants are linearly separable, and the line puts the missing quadrant on the wrong side.
05 / Uncertainty and trustWhat a confidence score is, and what it is not.
Aleatoric uncertainty is noise in the data; more data does not remove it. Epistemic uncertainty is ignorance of the model; more data can. A softmax conflates the two and is often sharp when it should be humble.
A classifier is calibrated if, among inputs where it predicts confidence p, the fraction that are correct is p:
The usual summary is the expected calibration error, which bins predictions and averages the confidence–accuracy gap:
Guo et al. (2017) showed that modern networks are systematically overconfident, and that one parameter helps: divide the logits by a temperature T before the softmax, and fit T on validation data. T > 1 softens, T < 1 sharpens. Temperature scaling does not change the class ranking, so accuracy is unchanged; only the numbers attached to the ranking change.
Calibration is distribution-dependent. Ovadia et al. (2019) found that under dataset shift both accuracy and calibration degrade, and most methods degrade with them. An iid calibration number is a statement about that set, not a per-input guarantee: a model can be calibrated on average and badly wrong on the subgroup that matters.
OOD, ensembles, and conformal sets
An OOD detector scores distance from the training distribution: maximum softmax probability, energy −log Σk exp(zk), or Mahalanobis distance in a feature layer. All need a threshold, and the threshold encodes a cost. “Out of distribution” is not one distribution: a detector tuned for fog may miss a new camera.
Ensembles train M models with different initializations; the mean prediction is usually better, and disagreement is a usable proxy for epistemic uncertainty. The cost is M times the compute, and shared bias makes the members agree and fail together. Conformal prediction instead gives a set with marginal coverage at least 1 − α under exchangeability — average over inputs, not per input.
Adversarial examples and underspecification
An adversarial example is an input changed by a small amount that flips the output. The fast gradient sign method is the smallest version:
Iterative versions such as PGD take smaller steps. The perturbations are often imperceptible and the wrong answer confident. Ilyas et al. (2019) argue that vulnerability is not only an optimization bug: models can rely on non-robust but genuinely predictive features, so the attack exploits a real correlation a human does not use. That makes feature admissibility a specification question.
Underspecification is the complementary finding: many pipelines with nearly identical iid accuracy behave very differently out of distribution, and preprocessing, hyperparameters, or the seed decide which behaviour you get (D'Amour et al., 2020). The model is the whole pipeline, not the architecture.
Interpretability is not explainability
Interpretability is the degree to which a human can understand why a system produced an output. Explainability usually means a post-hoc account: a saliency map, LIME, SHAP. A saliency map can be unstable and nearly unchanged when the parameters are randomized (Kindermans et al., 2017), which makes it a poor certificate. Rudin (2019) argues for inherently interpretable models in high-stakes decisions; an explanation that is not faithful transfers unearned trust.
| Question | Source | Estimator | What it does not tell you |
|---|---|---|---|
| Is the label noisy? | Aleatoric | Predictive entropy | Whether the model is wrong systematically |
| Is this region unfamiliar? | Epistemic | Ensemble disagreement; OOD score | Whether the input is answerable |
| Does confidence match accuracy? | Calibration | Reliability diagram; ECE | Whether it survives a shift |
| Is the answer guaranteed? | Coverage | Conformal prediction sets | Conditional coverage; score validity |
A confidence score summarises a model's behaviour on a distribution. It is not evidence that this answer is true, and it is not a safety case. If an application needs a guarantee, it has to come from something else: a specification, a monitor, a proof, or a statistical procedure with stated assumptions. This is a position about what “trust” should mean, not a theorem.
06 / What nets get rightThe balance sheet, and the bitter lesson.
The neural approach has won perception at scale (LeNet to AlexNet to ResNet), sequence modelling (the Transformer), transfer from pretrained representations, generation (VAEs, GANs, diffusion), and the engineering stack of autodiff and accelerators.
The failures are equally specific: opacity (the learned program is a matrix, not a text), data hunger, hallucination (likely text, not verified text), brittleness under shift and attack, composition, and long-horizon state. They cluster where the training objective and the deployment task diverge.
The strongest argument for the neural approach is historical, and it has a name. Sutton's “Bitter Lesson” (2019) states it in one sentence:
“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”
The evidence is a pattern: methods that encode human knowledge plateau; methods that search or learn at scale keep improving as compute improves.
But read it precisely. It is a claim about capability curves under increasing compute. It does not say that structure never helps; it says that structure which does not scale tends to lose to structure which does. It does not say that a scaled model is calibrated, auditable, data-efficient, or safe. A method can win the benchmark and still be unable to say why, or to guarantee that it will not do the wrong thing in a case you care about. This is a position, not a theorem: the bitter lesson is an empirical trend, and trends can be interrupted by a constraint the trend does not measure — energy, data, latency, regulation, or the cost of a wrong answer.
That is the gap the rest of this course lives in. Scale buys perception and fluency. It does not, by itself, buy composition, verification, data efficiency, or grounding. The symbolic world is a way to add those without pretending the neural world is unnecessary.
07 / The bridge to NeSyGrounders, proposers, and the other half of a check.
A network is very good at what symbolic systems are bad at: turning raw, noisy input into a structured object. The grounding role is an encoder φθ: X → Z followed by a quantizer q: Z → Σ into a symbol alphabet. The composition q ∘ φθ is a grounding function; its errors are the grounding errors of Lecture 05.
The second role is proposer: the network generates candidates — parses, plans, proofs, programs, actions — and a symbolic component checks them. Propose, check, learn from the check. AlphaGeometry is the clean case: a language model proposes auxiliary constructions, a deduction engine verifies them, and search continues until a proof is found.
The third role is constraint. A symbolic constraint restricts the hypothesis space, which is exactly an inductive bias. A grammar can mask invalid tokens during decoding:
This is exact only when the constraint is prefix-decomposable: whether a prefix can still be completed must be decidable from the prefix alone. For a grammar or a set of allowed tool calls it usually is; for a whole-output constraint it may not be, and masking can rule out the only valid completion. A wrong grammar makes the model confidently wrong in a legal-looking way.
These three roles connect to the five interfaces of Lecture 01: loss, architecture, data, search, verifier. A system can use several at once.
What does the interface add that scale does not? Four things, each with a price.
- Composition that survives recombination. A program is reusable: filter(cube, ·) works in a scene it has never seen. The price is a parser and a vocabulary that cover the domain.
- An independent check. A verifier does not share the network's biases, so it can catch an error the network is confident about. The price is that the verifier must be right.
- Data efficiency. A rule is a prior a finite sample would otherwise have to supply. The price is that a wrong rule is applied systematically.
- An audit trail. A derivation or a plan can be inspected, disputed, and repaired. The price is that it is only meaningful if the symbols are grounded and the rules sound.
Lecture 01's Sudoku lab is the interface in miniature. A neural perception module supplies a probability for each digit; the row, column, and box constraints decide which assignment is valid. At low noise the rules are almost redundant. At high noise they can repair a misread — or choose a different valid grid, because four clues do not identify which read was wrong. A symbolic component can only correct what its specification and its information allow. Open the Sudoku lab.
The honest case is not that symbols are more interpretable in the abstract. It is that a specific interface buys a specific property — composition, a check, data efficiency, an audit — at a specific cost: specification effort, inference time, semantic loss at the discretization, and the risk of trusting a wrong rule. The design question is which failure the application can afford.
08 / RecapWhat you should be able to do now.
Prove that no linear classifier computes XOR, and construct a hidden layer that does.
Explain a representation as a choice of what to ignore; name what the XOR construction discards.
Derive one gradient step by hand, and explain why forward activations are cached.
Read an architecture as an inductive bias: locality, sharing, recurrence, attention, equivariance.
Distinguish sufficiency, invariance, smoothness, and linear accessibility.
Separate iid, OOD, compositional, and causal generalization; explain double descent and its caveats.
Explain calibration, ECE, temperature scaling, and ensemble disagreement — and why none guarantees truth.
State what a symbolic interface adds and what it costs, without claiming it wins by default.
Exercises
- Prove it again. Write the four-inequality proof without looking at §0. Then fit XOR with features (x1, x2, x1x2) and find an explicit separator. What did the extra feature buy?
- Two step sizes. Redo the worked example with η = 0.05 and η = 1. Compute the loss after one step in each case, and explain the difference using the descent bound.
- Count the parameters. A dense layer maps a 64×64×3 image to 128 units; a 3×3 convolution maps 3 channels to 128. Compute both counts and the ratio. Which prior did the convolution add?
- Design a failure for the XOR representation. The map discards x1 + x2. Construct a labelling of the four points that this representation cannot solve with a linear readout, and prove it.
- Read a calibration claim. A model reports ECE = 0.03 on iid test data and 70% accuracy on a shifted set. What does the ECE still tell you, and what does it not? Rewrite the claim honestly.
Check your understanding
Ten questions, with explanations. The score is not the point; the questions you cannot answer are the reading list.
Reading list
The first six are the spine: backpropagation, the architectures, and the generalization puzzle. The rest are the caveats and the interface.
Learning representations by back-propagating errors
Rumelhart, Hinton & Williams · Nature 323:533–536, 1986 · the chain rule at scale
Gradient-based learning applied to document recognition
LeCun, Bottou, Bengio & Haffner · Proceedings of the IEEE 86(11):2278–2324, 1998 · LeNet
ImageNet classification with deep convolutional neural networks
Krizhevsky, Sutskever & Hinton · NeurIPS 2012; CACM reprint 2017 · AlexNet
Deep Residual Learning for Image Recognition
He, Zhang, Ren & Sun · CVPR 2016 · arXiv:1512.03385 · depth that trains
Attention Is All You Need
Vaswani et al. · NeurIPS 2017 · arXiv:1706.03762 · the Transformer
Understanding deep learning requires rethinking generalization
Zhang, Bengio, Hardt, Recht & Vinyals · ICLR 2017 · arXiv:1611.03530 · random labels
Learning Transferable Visual Models From Natural Language Supervision
Radford et al. · 2021 · arXiv:2103.00020 · CLIP and zero-shot transfer
Reconciling modern machine learning practice and the bias-variance trade-off
Belkin, Hsu, Ma & Mandal · PNAS 2019 · arXiv:1812.11118 · double descent
Shortcut Learning in Deep Neural Networks
Geirhos et al. · Nature Machine Intelligence 2020 · arXiv:2004.07780 · the cue that is not the task
In Search of Lost Domain Generalization
Gulrajani & Lopez-Paz · ICLR 2021 · arXiv:2007.01434 · DomainBed
On Calibration of Modern Neural Networks
Guo, Pleiss, Sun & Weinberger · ICML 2017 · arXiv:1706.04599 · temperature scaling
Can You Trust Your Model's Uncertainty?
Ovadia et al. · NeurIPS 2019 · arXiv:1906.02530 · calibration under shift
Adversarial Examples Are Not Bugs, They Are Features
Ilyas et al. · NeurIPS 2019 · arXiv:1905.02175 · non-robust features
The Bitter Lesson
Sutton · 2019 · the scale argument, in its own words
The worked gradient step and the parameter counts are exact arithmetic. Figures 0.1, 4.1, and 5.1 are schematics drawn by us and carry no measured values. The gallery diagrams are drawn by us, not reproduced. No benchmark scores are quoted. Where a claim is a position — the reading of the bitter lesson, the trust argument, the interface trade — the text says so.