Symbolic regression: the front, not the answer.
Sixteen points from a hidden expression. Enumerate 7,626 expressions, fit their constants, keep the Pareto front, and move the complexity penalty. Watch the selected expression change while the data stay the same.
The symbolic-regression lab
Legend. Teal dots are the data. The orange curve is the selected expression evaluated on the same x-range. On the front, lower-left is better: lower error and lower complexity. The highlighted point is the one the current penalty selects.
The selected expression
Score = RMSE + λ × complexity. The penalty is a prior about simplicity, not a fact. At λ = 0 the search selects the lowest error; at a larger λ it will pay for a shorter expression.
The front
| C | RMSE | Expression |
|---|
Reveal the generating expression
The data were generated by y = 2 sin(x) + 0.5 on 16 equally spaced points in [−3, 3], with no noise. The search can recover it because the grammar contains sine, addition, and constants, and because the constants grid contains 2 and 0.5. That is a luxury real data rarely provide.
Trace
Three choices, one dataset.
Enumeration by complexity
Every expression up to complexity 9 is generated, its constants are fitted by two coordinate-descent passes over a fixed grid, and its RMSE is computed. The best expression at each complexity is kept; the trace lists all nine. There is no sampling and no random seed.
λ selects, the data do not
At λ = 0.20 the score selects the exact generating expression at complexity 6. At λ = 0.35 it selects sin(x) at complexity 2; at λ = 0.60 it selects x at complexity 1. The data are identical in all three cases. The penalty is the modeling choice.
Pareto is not a ranking
Four points survive: C1, C2, C4, and C6. C4 is on the front, but no linear penalty selects it: it lies above the line between C2 and C6, so a linear score never prefers it. A front is a set of trade-offs, not an order. The selected point depends on the scalarization.
It is not a benchmark, a real scientific discovery, or a proof that the selected expression is true. The data are noise-free, the grammar contains the generating expression, the constants are fitted on a fixed grid, and the complexity measure is the number of nodes in the tree. It isolates one mechanism: enumeration, constant fitting, a complexity penalty, and a Pareto front. Real symbolic regression has noise, a grammar that may not contain the law, and a check that must be defended outside the training data.
Read the symbolic-regression section again · Read the five gaps · Slides