Course / Lecture 03 / Lab Lab 03

The representation playground.

Train a tiny network in the browser. Watch a linear model fail on XOR, a hidden layer solve it, and the same representation fail a compositional split while staying confident.

2 datasets 2 models 8 hidden units No install · runs in the browser

The representation lab

class 0 · train class 1 · train test · ring held-out · square OOD probe · cross
Ready.
epoch 0 / 420

How to read the board. The shading is the model's class probability; the dark line is the 0.5 boundary. Filled dots are training points, rings are iid test points, squares are the compositional held-out quadrant, and crosses are the OOD probes. The board is a model of the model, not the world.

Dataset

Model

14%

Jitter is Gaussian noise on the coordinates, not label noise. Changing it rebuilds the sample and resets the model.

Training metrics

—train acc
—test acc
—held-out

Loss

train loss —iid test loss —full-batch GD · η = 0.15

The teal curve is training loss; the orange curve is the held-out set (the compositional quadrant when it exists, otherwise the iid test set). The gap is the part a single accuracy number hides.

Hidden representation

Hidden activations projected on the first two principal components.

For the one-hidden-layer model this is a 2D view of an 8-dimensional activation space. If the two classes separate here, a linear readout can finish the job. For the linear model there is no hidden layer, so the panel shows the input space.

Out-of-distribution probes

pointp(class 1)confidencecall

Confidence is max(p, 1−p). This is not a calibration measurement; it is a per-input score, and it is not the probability that the prediction is correct.

Confidence

Train the model to read its confidence.

What to compare. If the held-out confidence is high while the held-out accuracy is low, the model is confidently wrong. That combination is the reason a confidence score is not a warrant.

WHAT THIS SHOWS

Three regimes, one mechanism.

REGIME 01 · XOR

A line stalls; a hidden layer fits

With the linear model, accuracy stays near chance: no line separates the diagonals. With one hidden layer, the representation folds the square and the readout separates the classes. This is the lecture's hook, trained rather than proved.

REGIME 02 · COMPOSITION

High iid accuracy, zero on the held-out corner

Switch the dataset to the compositional hold-out. The model sees three quadrants and is tested on the fourth. It can fit the iid test set perfectly and still put the missing combination on the wrong side, because the rule that covers it was never forced by the data.

REGIME 03 · CONFIDENCE

Confidently wrong

The held-out points are far from the training region. The model's confidence there can stay high while its accuracy is zero. A softmax is a statement about the model, not a measurement of the world.

What it does not show

It is a 2D toy with 8 tanh units, full-batch gradient descent, one fixed seed, and a synthetic distribution. It does not benchmark a real architecture, and it does not prove that compositional generalization is impossible. It shows a mechanism: the objective and the data decide which distinctions the representation keeps, and a split can expose a failure that iid accuracy hides. The PCA view is a 2D projection of an 8D space, so it can hide information as well as reveal it.

Back to generalization · Read the representation section · Open the slides