A monolithic vision-language model can learn to answer questions like this. But it cannot show you the steps, and it tends to fail when the steps are recombined in a way it has not seen.
The interesting part is not the answer. It is the machinery that makes the answer necessary.
S/V · Lecture 01
00 · The hookLecture 01 · What is neurosymbolic AI?
The Neuro-Symbolic Concept Learner
A scene graph, a program, an answer.
NS-CL learns concepts and parsing from question–answer pairs, without program annotations. After Mao et al. (arXiv:1904.12584).
The neural part grounds red, cube, and below in the scene. The symbolic part composes them. The program is not an explanation added after the fact; it is the computation.
S/V · Lecture 01
00 · The hookLecture 01 · What is neurosymbolic AI?
Interactive · the program, one step at a time
Run it yourself.
CLEVR dataset (Johnson et al.), CC BY 4.0 · arXiv:1612.06890
filter(red, scene)
filter(cube, ·)
below(large_red_sphere, ·)
count(·)
?
Press filter(red) to run the first step. Each step narrows the set.
S/V · Lecture 01
00 · The hookLecture 01 · What is neurosymbolic AI?
What just happened
The interface is the idea.
Neural
Perception, robustness, scaling from data. Strong where the input is noisy and the world is open.
Symbolic
Composition, deduction, verification. Strong where the answer must follow, and be checkable.
The interesting engineering is neither component. It is the seam: the loss, the executor, the data structure, the search, or the monitor that makes them one system.
S/V · Lecture 01
01 · The gapLecture 01 · What is neurosymbolic AI?
The gap · pure neural
Fluent, confident, not accountable.
Composition degrades with depth. Transformer performance on multi-step compositional tasks falls as the reasoning chain grows; the model relies on patterns seen in training. Faith and Fate, arXiv:2305.18654
Plans are plausible, not executable. LLMs can generate candidates, but need external critics and verifiers to check them. LLM-Modulo, arXiv:2402.01817
Embodied agents hallucinate the model. Preconditions, effects, and object grounding fail in characteristic ways. Embodied Agent Interface, arXiv:2410.07166
S/V · Lecture 01
01 · The gapLecture 01 · What is neurosymbolic AI?
The gap · pure symbolic
Correct, brittle, expensive to feed.
Someone has to write the model. Predicates, axioms, preconditions, effects. The knowledge-acquisition bottleneck is not a historical accident; it is the cost of formal representation.
The frame problem. A formal model must represent what changes, and implicitly what does not. The “does not” is enormous. McCarthy & Hayes, 1969
The perceptual boundary. A planner expects on(cup, table). It does not know what to do with a smudged image, an occlusion, or a novel category.
S/V · Lecture 01
01 · The gapLecture 01 · What is neurosymbolic AI?
The bet
Make the seam a first-class object.
Use each component where it is strong, and make the coupling explicit enough to inspect, train, and check. The bet is not always won. The rest of the course is about when it is, and what it costs.
continuous ↔ discrete
S/V · Lecture 01
02 · The formulationLecture 01 · What is neurosymbolic AI?
02 / 08
What exactly is a neurosymbolic model?
Three canonical forms, five interfaces, and the cost of each one.
S/V · Lecture 01
02 · The formulationLecture 01 · What is neurosymbolic AI?
A workable formulation
Three parts, three canonical forms.
Constrained prediction
ŷ = argmaxy pθ(y | x) s.t. K ⊨ y
The neural model scores; the symbolic component removes what violates the knowledge base.
Latent symbolic variable
p(y | x, K) = Σz pθ(z | x) · pR(y | z, K)
The neural part proposes symbolic structures; the symbolic part reasons over them.
Regularized learning
θ* = argminθ Ldata(θ) + λ Llogic(θ)
The constraint becomes a penalty during training, not a filter after it.
A neurosymbolic system has a neural component, a symbolic component, and an interface. The interface is what the three forms differ on.
S/V · Lecture 01
02 · The formulationLecture 01 · What is neurosymbolic AI?
The taxonomy this course uses
Five ways to cross the seam.
Interface
What it does
Canonical systems
Cost
Logic as loss
Compiles constraints into a differentiable penalty
Semantic loss, LTN, DL2
Hard constraints become soft
Logic as architecture
Puts a symbolic executor inside the model
DeepProbLog, NeurASP, Scallop
Inference cost; gradient quality
Logic as data
Uses symbols to structure inputs or embeddings
KG and ontology embeddings
Embedding distortion
Logic as search
Neural proposals plus symbolic search
ILP, AlphaGeometry, Lean
Search cost; proposal quality
Logic as verifier
Neural action plus a symbolic monitor
Shielding, STL, conformal bounds
Model fidelity; coverage
Pedagogical categories, not a claim of consensus. A system can use more than one.
S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?
What it buys: the model is trained to satisfy the constraints, not just checked after the fact. What it costs: satisfaction becomes a penalty, and independence assumptions can distort the semantics.
S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?
Interface 02 · architecture
Logic as an architecture.
DeepProbLog, NeurASP, DeepStochLog, Scallop. The logic is part of the computation.
What it buys: exact semantics and compositional structure. What it costs: inference is expensive, and gradients through discrete choices are hard.
S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?
Interface 03 · data
Logic as data.
Knowledge-graph and ontology embeddings; rule mining. The structure is in the data.
What it buys: interpretable structure and knowledge transfer. What it costs: the embedding can distort the structure, and the schema can drift.
S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?
Interface 04 · search
Logic as search.
ILP, program synthesis, AlphaGeometry, Lean. The strongest guarantees live here.
What it buys: compositional generalization and checkable outputs. What it costs: search can be expensive, and the neural proposal has to be good enough to make it tractable.
S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?
Interface 05 · verifier
Logic as a verifier.
Runtime monitoring, STL, conformal bounds, shielding. The logic can intervene.
What it buys: safety and accountability after the neural policy has acted. What it costs: the guarantee is only as good as the model and the specification.
S/V · Lecture 01
03 · The mapLecture 01 · What is neurosymbolic AI?
03 / 08
Where does the system sit?
Four axes, one tuple, and seven systems read along the same coordinates.
S/V · Lecture 01
03 · The mapLecture 01 · What is neurosymbolic AI?
Where the systems sit
A region, not a point.
Illustrative placement, not a measurement. Four axes: representation, inference, coupling, guarantee.
S/V · Lecture 01
03 · The mapLecture 01 · What is neurosymbolic AI?
A better answer than a label
Describe a system as a tuple.
(representation, inference, coupling, guarantee)
System
Representation
Inference
Coupling
Guarantee
Expert system
Symbolic
Deduction
Pipeline
Rule-level
DeepProbLog
Mixed
Probabilistic deduction
Architecture
Empirical
AlphaGeometry
Mixed
Deduction
Search
Formal proof
VLA policy
Neural
Reactive
None
None
If you cannot fill a slot, the system has not made that choice yet. The tuple is a design checklist.
S/V · Lecture 01
04 · History & philosophyLecture 01 · What is neurosymbolic AI?
Before the machines
The arguments are older than the models.
c. 350 BCE
Aristotle, Prior Analytics. Syllogistic logic; the first formal account of valid inference.
1679
Leibniz, calculus ratiocinator. A universal calculus of reasoning, unpublished in his lifetime.
1879
Frege, Begriffsschrift. Predicate logic with quantifiers; the logicist program.
1931
Gödel, incompleteness. Every sufficiently expressive formal system has true statements it cannot prove.
1936
Turing, computability. The universal machine; the limits of mechanical procedure.
1943
McCulloch & Pitts. A logical calculus of neural activity; neurons as threshold units.
1956
Dartmouth; Logic Theorist. Symbolic AI becomes a research program with a demo.
S/V · Lecture 01
04 · History & philosophyLecture 01 · What is neurosymbolic AI?
The camps and the synthesis
Two traditions, one unresolved question.
1958–59
McCarthy: LISP and common sense. Representation and commonsense reasoning become the agenda.
1969
Two critiques.Perceptrons exposes the limits of single-layer networks; the frame problem exposes the limits of naive logic.
1970s–80s
Expert systems, then winter. MYCIN, DENDRAL, Cyc; rules encode expertise, but maintenance is expensive.
1986–88
Backpropagation; probabilistic reasoning. Connectionism returns; Pearl formalizes uncertainty.
1990
Harnad, the symbol grounding problem. How does a symbol acquire meaning from the world?
2012–17
AlexNet; transformers. Scale, data, and attention change what neural systems can do.
2019–24
The third wave — and its critics. Sutton’s bitter lesson; Garcez and Lamb’s synthesis; AlphaGeometry; LLMs with tools.
S/V · Lecture 01
04 · History & philosophyLecture 01 · What is neurosymbolic AI?
The philosophical inheritance
Four problems that will not go away.
Symbol grounding
Harnad, 1990. A symbol defined only in terms of other symbols is arbitrary. Something has to touch the world.
The Chinese Room
Searle, 1980. Manipulating symbols according to rules is not the same as understanding them. Syntax is not semantics.
The frame problem
McCarthy & Hayes, 1969. A model must represent what changes, and implicitly what does not. The “does not” is enormous.
Abduction
Peirce. Most explanation is inference to the best explanation, not deduction from premises. A system that only deduces cannot diagnose.
S/V · Lecture 01
04 · History & philosophyLecture 01 · What is neurosymbolic AI?
The counter-argument
The bar is not “symbols are interpretable.”
Scale and learning tend to win. NeSy has to show a concrete benefit: less data, better composition, or a guarantee that scale alone does not deliver.
Sutton, “The Bitter Lesson,” 2019 · Garcez & Lamb, “Neurosymbolic AI: the 3rd wave,” 2023
S/V · Lecture 01
05 · The problemsLecture 01 · What is neurosymbolic AI?
What NeSy claims to solve
Four claims, four caveats.
Problem
Mechanism
Evidence
Caveat
Data efficiency
Background knowledge constrains the hypothesis space
LTN, DeepProbLog, semantic loss
Wrong knowledge can hurt
Compositional generalization
Symbolic composition of learned concepts
NS-CL, CLEVR, program synthesis
Synthetic benchmarks
Long-horizon reasoning
Plans, state transitions, search
LLM+P, SayCan, LLM-Modulo
Requires an environment model
Interpretability
Symbolic intermediates
NS-VQA, Logic-LM, traces
An intermediate can be wrong
The caveat is part of the claim: a mechanism that buys one property can cost another. The next slide is the other half of this table.
S/V · Lecture 01
05 · The problemsLecture 01 · What is neurosymbolic AI?
What NeSy claims to solve
Four more claims.
Problem
Mechanism
Evidence
Caveat
Verification & safety
Constraints, monitors, proofs
Shielding, STL, conformal bounds
Only as good as the spec
Knowledge integration
Ontologies, rules, graphs
KG reasoning, ontology embeddings
Schema drift; maintenance
Grounding
Perception-to-symbol interfaces
NS-CL, NS-VQA, scene graphs
Perception errors propagate
Discovery
Search over hypotheses with a verifier
AlphaGeometry, ILP, synthesis
Verification is domain-specific
What NeSy does not solve: open-world perception, common sense, causal discovery, or scaling laws. It is a set of interface strategies, not a theory of intelligence.
S/V · Lecture 01
06 · Seven examplesLecture 01 · What is neurosymbolic AI?
The program defines the sum exactly. arXiv:1805.10872
NS-CL executes programs over a latent scene; DeepProbLog puts neural predicates inside a probabilistic logic program. Both use the architecture interface, but the symbolic component does different work.
S/V · Lecture 01
06 · Seven examplesLecture 01 · What is neurosymbolic AI?
Constraint satisfaction · the lab
Rules repair perception.
Four clues; 288 valid 4×4 grids; the rules rank the completions.
Neural: a classifier reads each clue, often with noise.
Symbolic: every row, column, and box contains 1–4 exactly once.
Interface: the probabilities rank the 288 candidates; the rules decide which is possible.
InteractiveOne row, one misread.
1334
Perception reads 1, 3, 3, 4. A row must contain 1–4 exactly once.