Lecture 01 · Framing

What is neurosymbolic AI?

A system that sees but cannot reason is blind in a different way. A system that reasons but cannot see is empty. This is the seam between them.

21 lectures in the arc 7 worked examples 1 interactive lab Self-paced
00 · Learning objectivesLecture 01 · What is neurosymbolic AI?

By the end of this lecture

Four things you can do.

01

Define a neurosymbolic system

Name its three parts, and the interface that makes them one system rather than a pipeline.

02

Name the five interfaces

Loss, architecture, data, search, verifier — and what each one buys and costs.

03

Place a system on the map

Use the tuple (representation, inference, coupling, guarantee) on any paper you open.

04

Separate claims from evidence

State the caveat before the claim, and ask what the benchmark actually tests.

S/V · Lecture 01
00 · The hookLecture 01 · What is neurosymbolic AI?

A question a child can answer

“How many red cubes are below the large red sphere?”

A CLEVR scene: a large red metallic sphere in the centre, a red metallic cube in the lower centre, a large cyan cylinder on the left, a purple cylinder, two large gold cubes, a small gold cube, and small blue and green spheres.
CLEVR dataset (Johnson et al.), CC BY 4.0 · arXiv:1612.06890

A monolithic vision-language model can learn to answer questions like this. But it cannot show you the steps, and it tends to fail when the steps are recombined in a way it has not seen.

The interesting part is not the answer. It is the machinery that makes the answer necessary.

S/V · Lecture 01
00 · The hookLecture 01 · What is neurosymbolic AI?

The Neuro-Symbolic Concept Learner

A scene graph, a program, an answer.

image object-centric scene graph count( filter(cube, filter(red, below(large_red_sphere)))) program 1 answer
NS-CL learns concepts and parsing from question–answer pairs, without program annotations. After Mao et al. (arXiv:1904.12584).

The neural part grounds red, cube, and below in the scene. The symbolic part composes them. The program is not an explanation added after the fact; it is the computation.

S/V · Lecture 01
00 · The hookLecture 01 · What is neurosymbolic AI?

Interactive · the program, one step at a time

Run it yourself.

The CLEVR scene used in the question: a red sphere, a red cube, a red cylinder, and other objects.
CLEVR dataset (Johnson et al.), CC BY 4.0 · arXiv:1612.06890
filter(red, scene)
filter(cube, ·)
below(large_red_sphere, ·)
count(·)
?

Press filter(red) to run the first step. Each step narrows the set.

S/V · Lecture 01
00 · The hookLecture 01 · What is neurosymbolic AI?

What just happened

The interface is the idea.

Neural

Perception, robustness, scaling from data. Strong where the input is noisy and the world is open.

Symbolic

Composition, deduction, verification. Strong where the answer must follow, and be checkable.

The interesting engineering is neither component. It is the seam: the loss, the executor, the data structure, the search, or the monitor that makes them one system.
S/V · Lecture 01
01 · The gapLecture 01 · What is neurosymbolic AI?

The gap · pure neural

Fluent, confident, not accountable.

Composition degrades with depth. Transformer performance on multi-step compositional tasks falls as the reasoning chain grows; the model relies on patterns seen in training. Faith and Fate, arXiv:2305.18654
Plans are plausible, not executable. LLMs can generate candidates, but need external critics and verifiers to check them. LLM-Modulo, arXiv:2402.01817
Embodied agents hallucinate the model. Preconditions, effects, and object grounding fail in characteristic ways. Embodied Agent Interface, arXiv:2410.07166
S/V · Lecture 01
01 · The gapLecture 01 · What is neurosymbolic AI?

The gap · pure symbolic

Correct, brittle, expensive to feed.

Someone has to write the model. Predicates, axioms, preconditions, effects. The knowledge-acquisition bottleneck is not a historical accident; it is the cost of formal representation.
The frame problem. A formal model must represent what changes, and implicitly what does not. The “does not” is enormous. McCarthy & Hayes, 1969
The perceptual boundary. A planner expects on(cup, table). It does not know what to do with a smudged image, an occlusion, or a novel category.
S/V · Lecture 01
01 · The gapLecture 01 · What is neurosymbolic AI?

The bet

Make the seam a first-class object.

Use each component where it is strong, and make the coupling explicit enough to inspect, train, and check. The bet is not always won. The rest of the course is about when it is, and what it costs.

S/V · Lecture 01
02 · The formulationLecture 01 · What is neurosymbolic AI?
02 / 08

What exactly is a neurosymbolic model?

Three canonical forms, five interfaces, and the cost of each one.

S/V · Lecture 01
02 · The formulationLecture 01 · What is neurosymbolic AI?

A workable formulation

Three parts, three canonical forms.

Constrained prediction

ŷ = argmaxy pθ(y | x)
s.t. K ⊨ y

The neural model scores; the symbolic component removes what violates the knowledge base.

Latent symbolic variable

p(y | x, K) =
Σz pθ(z | x) · pR(y | z, K)

The neural part proposes symbolic structures; the symbolic part reasons over them.

Regularized learning

θ* = argminθ
Ldata(θ) + λ Llogic(θ)

The constraint becomes a penalty during training, not a filter after it.

A neurosymbolic system has a neural component, a symbolic component, and an interface. The interface is what the three forms differ on.

S/V · Lecture 01
02 · The formulationLecture 01 · What is neurosymbolic AI?

The taxonomy this course uses

Five ways to cross the seam.

InterfaceWhat it doesCanonical systemsCost
Logic as lossCompiles constraints into a differentiable penaltySemantic loss, LTN, DL2Hard constraints become soft
Logic as architecturePuts a symbolic executor inside the modelDeepProbLog, NeurASP, ScallopInference cost; gradient quality
Logic as dataUses symbols to structure inputs or embeddingsKG and ontology embeddingsEmbedding distortion
Logic as searchNeural proposals plus symbolic searchILP, AlphaGeometry, LeanSearch cost; proposal quality
Logic as verifierNeural action plus a symbolic monitorShielding, STL, conformal boundsModel fidelity; coverage

Pedagogical categories, not a claim of consensus. A system can use more than one.

S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?

Interface 01 · loss

Logic as a loss.

neural model output logic penalty L = L_data + λ L_logic gradient back to θ
Semantic loss, Logic Tensor Networks, DL2. End-to-end training; soft constraints.

What it buys: the model is trained to satisfy the constraints, not just checked after the fact. What it costs: satisfaction becomes a penalty, and independence assumptions can distort the semantics.

S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?

Interface 02 · architecture

Logic as an architecture.

neural predicate p(digit | image) logic program add(X,Y,Z) distribution over answers train
DeepProbLog, NeurASP, DeepStochLog, Scallop. The logic is part of the computation.

What it buys: exact semantics and compositional structure. What it costs: inference is expensive, and gradients through discrete choices are hard.

S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?

Interface 03 · data

Logic as data.

query graph embedding similarity · structure rules constraints · trace answer
Knowledge-graph and ontology embeddings; rule mining. The structure is in the data.

What it buys: interpretable structure and knowledge transfer. What it costs: the embedding can distort the structure, and the schema can drift.

S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?

Interface 04 · search

Logic as search.

neural proposal candidate · heuristic symbolic search deduction · verification check pass / fail proof or plan
ILP, program synthesis, AlphaGeometry, Lean. The strongest guarantees live here.

What it buys: compositional generalization and checkable outputs. What it costs: search can be expensive, and the neural proposal has to be good enough to make it tractable.

S/V · Lecture 01
02 · The five interfacesLecture 01 · What is neurosymbolic AI?

Interface 05 · verifier

Logic as a verifier.

neural policy proposed action monitor / shield spec · STL · bound allow override safe action
Runtime monitoring, STL, conformal bounds, shielding. The logic can intervene.

What it buys: safety and accountability after the neural policy has acted. What it costs: the guarantee is only as good as the model and the specification.

S/V · Lecture 01
03 · The mapLecture 01 · What is neurosymbolic AI?
03 / 08

Where does the system sit?

Four axes, one tuple, and seven systems read along the same coordinates.

S/V · Lecture 01
03 · The mapLecture 01 · What is neurosymbolic AI?

Where the systems sit

A region, not a point.

THE SEAM SYMBOLIC NEURAL INDUCTION / ABDUCTION DEDUCTION Expert systems Description logics Classical planning ILP Program synthesis Bayesian networks DeepProbLog LTN NS-CL AlphaGeometry Logic-LM / LINC SayCan VLA policies Deep learning
Illustrative placement, not a measurement. Four axes: representation, inference, coupling, guarantee.
S/V · Lecture 01
03 · The mapLecture 01 · What is neurosymbolic AI?

A better answer than a label

Describe a system as a tuple.

(representation, inference, coupling, guarantee)

SystemRepresentationInferenceCouplingGuarantee
Expert systemSymbolicDeductionPipelineRule-level
DeepProbLogMixedProbabilistic deductionArchitectureEmpirical
AlphaGeometryMixedDeductionSearchFormal proof
VLA policyNeuralReactiveNoneNone
If you cannot fill a slot, the system has not made that choice yet. The tuple is a design checklist.
S/V · Lecture 01
04 · History & philosophyLecture 01 · What is neurosymbolic AI?

Before the machines

The arguments are older than the models.

c. 350 BCE
Aristotle, Prior Analytics. Syllogistic logic; the first formal account of valid inference.
1679
Leibniz, calculus ratiocinator. A universal calculus of reasoning, unpublished in his lifetime.
1879
Frege, Begriffsschrift. Predicate logic with quantifiers; the logicist program.
1931
Gödel, incompleteness. Every sufficiently expressive formal system has true statements it cannot prove.
1936
Turing, computability. The universal machine; the limits of mechanical procedure.
1943
McCulloch & Pitts. A logical calculus of neural activity; neurons as threshold units.
1956
Dartmouth; Logic Theorist. Symbolic AI becomes a research program with a demo.
S/V · Lecture 01
04 · History & philosophyLecture 01 · What is neurosymbolic AI?

The camps and the synthesis

Two traditions, one unresolved question.

1958–59
McCarthy: LISP and common sense. Representation and commonsense reasoning become the agenda.
1969
Two critiques. Perceptrons exposes the limits of single-layer networks; the frame problem exposes the limits of naive logic.
1970s–80s
Expert systems, then winter. MYCIN, DENDRAL, Cyc; rules encode expertise, but maintenance is expensive.
1986–88
Backpropagation; probabilistic reasoning. Connectionism returns; Pearl formalizes uncertainty.
1990
Harnad, the symbol grounding problem. How does a symbol acquire meaning from the world?
2012–17
AlexNet; transformers. Scale, data, and attention change what neural systems can do.
2019–24
The third wave — and its critics. Sutton’s bitter lesson; Garcez and Lamb’s synthesis; AlphaGeometry; LLMs with tools.
S/V · Lecture 01
04 · History & philosophyLecture 01 · What is neurosymbolic AI?

The philosophical inheritance

Four problems that will not go away.

Symbol grounding

Harnad, 1990. A symbol defined only in terms of other symbols is arbitrary. Something has to touch the world.

The Chinese Room

Searle, 1980. Manipulating symbols according to rules is not the same as understanding them. Syntax is not semantics.

The frame problem

McCarthy & Hayes, 1969. A model must represent what changes, and implicitly what does not. The “does not” is enormous.

Abduction

Peirce. Most explanation is inference to the best explanation, not deduction from premises. A system that only deduces cannot diagnose.

S/V · Lecture 01
04 · History & philosophyLecture 01 · What is neurosymbolic AI?

The counter-argument

The bar is not “symbols are interpretable.”

Scale and learning tend to win. NeSy has to show a concrete benefit: less data, better composition, or a guarantee that scale alone does not deliver.

Sutton, “The Bitter Lesson,” 2019 · Garcez & Lamb, “Neurosymbolic AI: the 3rd wave,” 2023

S/V · Lecture 01
05 · The problemsLecture 01 · What is neurosymbolic AI?

What NeSy claims to solve

Four claims, four caveats.

ProblemMechanismEvidenceCaveat
Data efficiencyBackground knowledge constrains the hypothesis spaceLTN, DeepProbLog, semantic lossWrong knowledge can hurt
Compositional generalizationSymbolic composition of learned conceptsNS-CL, CLEVR, program synthesisSynthetic benchmarks
Long-horizon reasoningPlans, state transitions, searchLLM+P, SayCan, LLM-ModuloRequires an environment model
InterpretabilitySymbolic intermediatesNS-VQA, Logic-LM, tracesAn intermediate can be wrong
The caveat is part of the claim: a mechanism that buys one property can cost another. The next slide is the other half of this table.
S/V · Lecture 01
05 · The problemsLecture 01 · What is neurosymbolic AI?

What NeSy claims to solve

Four more claims.

ProblemMechanismEvidenceCaveat
Verification & safetyConstraints, monitors, proofsShielding, STL, conformal boundsOnly as good as the spec
Knowledge integrationOntologies, rules, graphsKG reasoning, ontology embeddingsSchema drift; maintenance
GroundingPerception-to-symbol interfacesNS-CL, NS-VQA, scene graphsPerception errors propagate
DiscoverySearch over hypotheses with a verifierAlphaGeometry, ILP, synthesisVerification is domain-specific
What NeSy does not solve: open-world perception, common sense, causal discovery, or scaling laws. It is a set of interface strategies, not a theory of intelligence.
S/V · Lecture 01
06 · Seven examplesLecture 01 · What is neurosymbolic AI?
06 / 08

One anatomy, seven domains.

Vision, probabilistic logic, constraints, mathematics, robotics, language, knowledge graphs.

S/V · Lecture 01
06 · Seven examplesLecture 01 · What is neurosymbolic AI?

Vision and probabilistic logic

Same anatomy, different interface.

NS-CL · vision

Neural: object-centric perception.
Symbolic: programs over the scene graph.
Interface: architecture / execution.
Guarantee: empirical.

The program is the explanation. arXiv:1904.12584

DeepProbLog · logic

Neural: digit classifier.
Symbolic: probabilistic logic program.
Interface: neural predicates.
Guarantee: empirical.

The program defines the sum exactly. arXiv:1805.10872

NS-CL executes programs over a latent scene; DeepProbLog puts neural predicates inside a probabilistic logic program. Both use the architecture interface, but the symbolic component does different work.

S/V · Lecture 01
06 · Seven examplesLecture 01 · What is neurosymbolic AI?

Constraint satisfaction · the lab

Rules repair perception.

1432 23 3412 2143 41
Four clues; 288 valid 4×4 grids; the rules rank the completions.

Neural: a classifier reads each clue, often with noise.

Symbolic: every row, column, and box contains 1–4 exactly once.

Interface: the probabilities rank the 288 candidates; the rules decide which is possible.

InteractiveOne row, one misread.
1 3 3 4

Perception reads 1, 3, 3, 4. A row must contain 1–4 exactly once.

NeurASP, arXiv:2307.07700 · DeepProbLog, arXiv:1805.10872

S/V · Lecture 01
06 · Seven examplesLecture 01 · What is neurosymbolic AI?

Mathematics · search + proof

A model proposes; a prover disposes.

problem + auxiliary construction
The language model proposes constructions; the deduction engine checks the proof.

Neural: a language model trained on synthetic proofs.

Symbolic: a deduction engine that checks every step.

Interface: search — the model guides, the engine verifies.

Guarantee: a formal proof, not a plausible sentence.

Trinh et al. 2024 · Nature · DOI 10.1038/s41586-023-06747-5

S/V · Lecture 01
06 · Seven examplesLecture 01 · What is neurosymbolic AI?

Robotics, language, knowledge

Three more interfaces.

SayCan · robotics

Neural: LLM + value functions.
Symbolic: skill affordances.
Interface: search / feasibility.
Guarantee: empirical.

Language supplies the plan; affordances supply the grounding. arXiv:2204.01691

Logic-LM · language

Neural: LLM formalizer.
Symbolic: solver.
Interface: tool use + repair.
Guarantee: empirical / formal.

Translate, solve, feed errors back. arXiv:2305.12295

KG reasoning · knowledge

Neural: graph embeddings.
Symbolic: rules and constraints.
Interface: data.
Guarantee: empirical.

Structure as a prior, with a trace. arXiv:2302.07200

The pattern repeats: a neural proposal, a symbolic constraint, and a guarantee that is only as good as the model.

S/V · Lecture 01
07 · The courseLecture 01 · What is neurosymbolic AI?

The 21-lecture arc

One spine, six acts.

I · Framing

What is NeSy? Definition, taxonomy, history, problems, seven examples.

II · The two worlds

Symbolic AI; neural AI; graphs and relations; the interface problems.

III · The five interfaces

Loss, architecture, data, search, verifier — one lecture each, plus a comparison.

IV · The three inferences

Deduction, induction, abduction with language models; agents and world models.

V · Applications

Vision, NLP/KG, science/math, robotics, and uncommon domains.

VI · Frontiers

Benchmarks, evaluation, and how to read a NeSy paper.

S/V · Lecture 01
07 · The courseLecture 01 · What is neurosymbolic AI?

A reading protocol

Five questions.

1 · What is the neural component? A classifier, an encoder, a language model, a policy, a value function. What does it map from and to?
2 · What is the symbolic component? A logic program, a set of constraints, a planner, a prover. What language, and what assumptions?
3 · What is the interface? Loss, architecture, data, search, or verifier. Where does information cross, and in which direction?
4 · What is the guarantee? Empirical, formal, or none. If formal, what exactly is proved?
5 · What does the benchmark test? Composition, grounding, data efficiency, planning — or something else.
S/V · Lecture 01
08 · RecapLecture 01 · What is neurosymbolic AI?

What you should be able to do

Four moves.

Define

A neural component, a symbolic component, and an explicit interface.

Place

Use the tuple: representation, inference, coupling, guarantee.

Name

The five interfaces: loss, architecture, data, search, verifier.

Separate

The claim from the evidence; the benchmark from the world.

S/V · Lecture 01
08 · RecapLecture 01 · What is neurosymbolic AI?

Next · hands-on

Now break it.

Open the Sudoku lab. Turn the noise up. Watch the rules repair a read — or choose a different valid world.

Lab 01 · Sudoku → open the lab

S/V · Lecture 01
Speaker notes