The representation playground.
Train a tiny network in the browser. Watch a linear model fail on XOR, a hidden layer solve it, and the same representation fail a compositional split while staying confident.
The representation lab
How to read the board. The shading is the model's class probability; the dark line is the 0.5 boundary. Filled dots are training points, rings are iid test points, squares are the compositional held-out quadrant, and crosses are the OOD probes. The board is a model of the model, not the world.
Dataset
Model
Jitter is Gaussian noise on the coordinates, not label noise. Changing it rebuilds the sample and resets the model.
Training metrics
Loss
train loss —iid test loss —full-batch GD · η = 0.15
The teal curve is training loss; the orange curve is the held-out set (the compositional quadrant when it exists, otherwise the iid test set). The gap is the part a single accuracy number hides.
Hidden representation
Hidden activations projected on the first two principal components.
For the one-hidden-layer model this is a 2D view of an 8-dimensional activation space. If the two classes separate here, a linear readout can finish the job. For the linear model there is no hidden layer, so the panel shows the input space.
Out-of-distribution probes
| point | p(class 1) | confidence | call |
|---|
Confidence is max(p, 1−p). This is not a calibration measurement; it is a per-input score, and it is not the probability that the prediction is correct.
Confidence
Train the model to read its confidence.
What to compare. If the held-out confidence is high while the held-out accuracy is low, the model is confidently wrong. That combination is the reason a confidence score is not a warrant.
Three regimes, one mechanism.
A line stalls; a hidden layer fits
With the linear model, accuracy stays near chance: no line separates the diagonals. With one hidden layer, the representation folds the square and the readout separates the classes. This is the lecture's hook, trained rather than proved.
High iid accuracy, zero on the held-out corner
Switch the dataset to the compositional hold-out. The model sees three quadrants and is tested on the fourth. It can fit the iid test set perfectly and still put the missing combination on the wrong side, because the rule that covers it was never forced by the data.
Confidently wrong
The held-out points are far from the training region. The model's confidence there can stay high while its accuracy is zero. A softmax is a statement about the model, not a measurement of the world.
It is a 2D toy with 8 tanh units, full-batch gradient descent, one fixed seed, and a synthetic distribution. It does not benchmark a real architecture, and it does not prove that compositional generalization is impossible. It shows a mechanism: the objective and the data decide which distinctions the representation keeps, and a split can expose a failure that iid accuracy hides. The PCA view is a 2D projection of an 8D space, so it can hide information as well as reveal it.
Back to generalization · Read the representation section · Open the slides