Course / Act IV · Composition Lecture 15

Agents and world models.

A robot sees a window, remembers a map, imagines a path, and moves. The agent is not one model but a loop closed through a model — and the model is where the course’s interfaces meet.

~50 min read 9 sections 3 world-model families 1 interactive lab Updated 2026-11-17

A mobile manipulator is in a kitchen. The instruction is one sentence: “Put the apples in the fridge.” The camera sees a five-by-five window. The fridge is behind the robot. Nothing in the image contains the word fridge, the path to it, or the fact that the door is closed. Yet the sentence has to become a sequence of torques.

00 / The hookThe map is not the world.

Break the sentence into questions. Which apples? That is grounding: mapping words to objects, places, and relations. Where is the fridge? That is memory and a spatial model, because the fridge is outside the view. What happens if I open the door? That is a world model: a prediction of the consequences of an action. In what order? That is planning: search over a model for a sequence that reaches the goal. How do I move the arm? That is motion planning and control: continuous, collision-aware, feedback-driven. How do I know it worked? That is monitoring: comparing prediction with observation and repairing the plan or the model when they disagree.

Each question is a topic from earlier in the course. An embodied agent is what you get when you put them in a loop and require the loop to close in real time. The pieces are not new; the engineering problem is the seam between them.

The one idea

An agent is a loop closed through a model. The model predicts what actions do; the loop compares predictions with observations and repairs whichever is wrong. A model without a loop is a prior. A loop without a model is a reflex. Most interesting failures happen at the seam between them.

Scope · two kinds of agent

“Agent” is overloaded. This lecture covers embodied decision makers (robots, simulators, grid worlds) and tool-using language agents (ReAct-style loops, LLM-Modulo). They share the loop and the model; they differ in what the model is made of and what counts as an action. This is about the composition, not a survey of RL.

Table 0.1 · The five stages of the loop, the question each answers, and the interface it uses.
StageQuestionWhat it producesCourse interface
PerceiveWhat is the world, given what I can see?A belief over states and relationsGrounding; logic as data
ModelWhat would happen if I did a?A prediction or a rolloutRepresentation; logic as architecture
PlanWhich sequence reaches the goal?A task plan, a path, or a policyLogic as search
ActWhat command do I send now?An action, trajectory, or torqueControl; architecture
MonitorDid the world do what I predicted?A mismatch signal and a repairLogic as verifier

The sections take the stages apart: §1 formalises the loop, §2 compares three world-model families, §3 separates the planning layers, §4 adds language, §5 memory and tools, §6 evaluation, and §7 failure modes. The lab lets you remove the model, or remove the loop, and watch.

01 / The agent loopA loop closed through a model.

The standard formalism is a partially observable Markov decision process (POMDP). A state s ∈ S is the part of the world that matters; an action a ∈ A changes it; an observation o ∈ O is what the agent senses. The agent does not know s. It maintains a belief bt(s) = P(st = s | o1:t, a1:t−1) and updates it after each action and observation. With a known transition model T(s′ | s, a) and observation model O(o | s′), the update is the Bayes filter:

bt+1(s′) = η · O(ot+1 | s′) · Σs T(s′ | s, at) · bt(s)
(1.1)

The sum is the predict step: push the previous belief through the transition model. The multiplication by the observation likelihood is the update step: reweight the prediction by how well each state explains what was seen. η normalises. The equation is simple; the difficulty is that T and O are usually learned, approximate, and drifting.

A policy maps a belief (or a history) to an action. A model-based agent queries a model at decision time and evaluates counterfactuals; a reactive agent maps the current observation directly to an action. The distinction is not “has a model in its weights” versus “has none” — a reactive policy can encode a great deal of structure. It is whether the agent queries a model of consequences when it decides. That is the difference the lab makes visible.

  • Perceive. A detector proposes objects and attributes; a symbolic layer asserts facts such as on(cup, table). The output is a claim with uncertainty, not a fact. Treating it as ground truth and blaming the planner is the most common design error in agent systems.
  • Model. A map, simulator, learned dynamics model, PDDL domain, language prior, or mixture. It answers “what would happen if?” for one action, a rollout, or a distribution. A model that cannot answer counterfactuals is a representation, not a world model.
  • Plan. Search or optimisation over the model for operators, a trajectory, a policy, or a contingency tree. Planning is the deliberate version; a learned policy is the compiled version. Only the first is inspectable at decision time.
  • Act. Execute one action, a short prefix, or the whole plan. No feedback is open loop; one step and re-plan is closed loop. The controller closes the fastest loop.
  • Monitor. Compare prediction with observation. A mismatch can trigger re-perception, re-planning, slower execution, or a model update. This is the inference step that keeps the loop honest.
The agent loop as five stages Perceive feeds a model; the model feeds a plan; the plan feeds an action; the action feeds a monitor; the monitor feeds back to perception and planning. The belief is the shared state. PERCEIVE sense · ground update belief MODEL predict · rollout latent / symbolic PLAN search · optimise task · motion ACT execute · control close the loop MONITOR compare · detect replan or revise
Fig. 1.1 The loop as a composition: each stage is an interface from an earlier lecture, and each arrow is a contract. The dashed return path is the monitor.
belief ← prior
plan ← ∅
repeat
  o ← sense()
  belief ← update(belief, o)            # grounding / perception
  if plan is empty or inconsistent(belief, plan):
      plan ← search(model(belief), goal)  # task + motion planning
  a ← first(plan)
  execute(a)                             # controller closes the inner loop
  predicted ← rollout(model(belief), a)
  if disagree(predicted, o):
      replan or revise(model)            # monitor / verifier
until goal(belief) or budget exhausted

The pseudo-code hides the key engineering choice: the rates. A controller runs at hundreds of hertz; perception at tens; task planning at a fraction of a hertz; model learning offline. A planner cannot run at servo rate, and a torque loop cannot close at planning rate.

Suppose a robot needs to know whether a door is open. Its prior is even: P(open) = 0.5. Its sensor reports “open” with probability 0.9 when the door is open, and 0.1 when it is closed. After one “open,” the posterior is (0.5 × 0.9) / (0.5 × 0.9 + 0.5 × 0.1) = 0.9; a second independent “open” gives 0.81 / 0.82 ≈ 0.988. Two weak observations can beat one strong one only if errors are independent — usually false for a real sensor. If route A is 6 steps when the door is open and 20 when it is closed, while route B is always 8, then A costs 0.9 × 6 + 0.1 × 20 = 7.4 in expectation. The numbers are a toy; the point is that a plan is a bet on a model.

Caveat · the filter is only as good as its models

Equation 1.1 assumes T and O are known and stationary. In a real agent they are learned, approximate, and drifting. Repeated observations of the same misleading cue do not create information if the errors are correlated. The loop needs a monitor that detects “the world is not behaving as my model says,” not only a filter that updates within the model.

02 / World modelsLatent, symbolic, hybrid.

A world model predicts the consequences of actions and scores outcomes. It can be a map, a simulator, logical operators, a learned dynamics network, or a language prior. The three families are different answers to “what is the state, and how does it change?” Each answer buys something at a cost.

Latent world models

A latent model learns an encoder zt = encθ(ot), a transition zt+1 ~ pθ(zt+1 | zt, at), and heads for reward, value, or reconstruction. Planning happens in the latent space, not in pixels. The canonical factorisation is

pθ(zt+1, ot, rt | zt, at) = pθ(zt+1 | zt, at) · pθ(ot | zt) · pθ(rt | zt)
(2.1)

Ha and Schmidhuber’s World Models (arXiv:1803.10122) compressed frames into a VAE latent, predicted the next latent with an MDN-RNN, and evolved a controller inside the learned dream. Dreamer (Hafner et al., arXiv:1912.01603) learns the model from pixels and trains an actor-critic by backpropagating through imagined latent rollouts. MuZero (Schrittwieser et al., arXiv:1911.08265) learns a model that predicts reward, policy, and value, not observations, and uses it inside Monte Carlo tree search.

Latent models scale to high-dimensional observations and learn without a hand-written abstraction. The cost is opacity and distribution dependence: a model can reconstruct beautiful images and still be useless for control. MuZero’s “value-equivalent” model is a reminder that prediction accuracy is not planning usefulness.

Compounding error

Model-based planning multiplies one-step errors over the horizon. Let the true transition f and the learned model f̂ both be L-Lipschitz, with one-step error bounded by ε. An open-loop rollout of length H satisfies

‖fH(s) − f̂H(s)‖ ≤ ε · (1 + L + ⋯ + LH−1) = ε · (LH − 1) / (L − 1)
(2.2)

for L ≠ 1, and εH for L = 1. The bound telescopes: each step can add L times the previous error. It is worst-case and open-loop; a closed-loop agent re-observes and corrects. It explains why long-horizon planning is hard and why an optimiser can exploit the model’s optimistic mistakes.

Symbolic world models

A symbolic model represents state as facts and actions as operators with preconditions and effects. In the STRIPS convention (Fikes & Nilsson, 1971), an action a is applicable when its preconditions are a subset of the state; the successor deletes the delete-list and adds the add-list:

s′ = (s \ del(a)) ∪ add(a)    if   pre(a) ⊆ s
(2.3)

PDDL generalises this with typed objects, quantified goals, conditional effects, and durative actions (McDermott et al., 1998). A planner searches for a sequence that reaches the goal; the plan can be inspected, validated, repaired, and compared. The guarantee is conditional: a sound and complete planner achieves the goal in the model, not in the world.

Three classical problems mark the boundary. The frame problem (McCarthy & Hayes, 1969) is how to represent what does not change without enumerating every unaffected fact. The qualification problem is that preconditions are never exhaustively listable: a door can be locked, blocked, too heavy, or open inward. The ramification problem is that actions have indirect effects: moving a cup changes what is reachable, visible, and supporting something else. Explicitness is not completeness.

Hybrid world models

A hybrid model puts a learned representation and a symbolic structure together. Four patterns: object-centric perception plus symbolic relations (a detector builds a scene graph, as in NS-CL); a language model plus a classical planner (an LLM writes a PDDL problem, as in LLM+P, arXiv:2304.11477); learned skills plus a symbolic task layer (SayCan grounds a skill sequence in affordances); and learned dynamics plus symbolic constraints (a neural proposal checked by collision or kinematic constraints).

Hybrids relocate the interface problem. A detector labels a closed door as open; the fact is used; the plan is valid; the robot collides. Each component can be correct and the system still fails.

Table 2.1 · Three families of world model. The trade-offs are typical, not universal; real systems mix them.
DimensionLatentSymbolicHybrid
StateContinuous vector or distributionSet of facts, typed objectsObjects, attributes, relations
TransitionLearned network; stochasticOperators with preconditions and effectsLearned dynamics plus constraints
PlanningRollouts, MPC, gradient-basedSearch over operators; A*, PDDLTask planner over skills; verifier
Data efficiencyNeeds interaction dataNeeds a written modelNeeds both, usually less of each
InspectabilityLow; latent state is opaqueHigh; plan and operators are readableMedium; the seam is the opaque part
GuaranteeEmpirical, distribution-dependentSoundness relative to the modelAs good as the weakest component
Typical failureCompounding error; model exploitationMissing precondition; wrong initial stateGrounding error in a valid plan
Caveat · “model-based” is a spectrum

Model-free RL can be trained inside a model-based simulator; a model-based planner can use a model with no explicit reward. The useful distinction is whether the agent evaluates counterfactual action sequences at decision time. A policy that memorised a maze is not planning; a planner that queries a learned model is.

03 / Planning & controlThree layers, three time scales.

“Planning” is used for three problems with different state spaces, algorithms, success criteria, and failure modes. Confusing them produces bad evaluations: a system is said to “fail at planning” when the task plan was fine and the controller could not track it, or when the task plan was geometrically infeasible from the start.

Table 3.1 · The three layers. Timescales are orders of magnitude, not specifications.
LayerState spaceDecisionRateSuccessTypical failure
Task planningDiscrete facts and objects (state space)Which operator nextSeconds to minutesGoal formula satisfiedNo plan; wrong model; wrong goal
Motion planningContinuous configuration spaceCollision-free path0.1–10 sPath exists and is collision-freeInfeasible; narrow passage; dynamic obstacle
ControlContinuous state and commandsTorque or velocity command100–1000 HzStable tracking; bounded errorInstability; saturation; slip

Task planning searches over discrete operators. The input is a domain, an initial state, and a goal; the output is a sequence, a partial order, or a policy. Forward search with A* and a heuristic is the baseline; delete-relaxation heuristics such as hadd and hmax make it practical. A task plan can be validated without executing it: check each operator’s preconditions and the final goal. That is a real guarantee about the model. If the initial state says the door is open and it is closed, the validator still certifies the plan.

Motion planning searches a continuous configuration space for a path that avoids obstacles and satisfies kinematic constraints. Sampling-based planners (RRT, PRM) explore; trajectory optimisation refines. Both need a collision model, which is itself a world model. A path can be collision-free and dynamically infeasible: too fast for the motors, too tight for the gripper, or unreachable from the current base pose.

Control closes the fastest loop. A PID controller corrects tracking error; model predictive control (MPC) optimises a short horizon at each tick and executes the first action:

a*0:H−1 = argmina0:H−1 Esk+1 ∼ T(· | sk, ak) [ Σk=0H−1 c(sk, ak) + cf(sH) ]
(3.1)

Receding-horizon execution is the loop: optimise, execute one step, observe, optimise again.

A task plan is feasible only if the motion layer can realise each abstract action. For a task plan σ = (a1, …, an) and trajectories ξi, the plan is realisable when the trajectories connect, stay collision-free, and end in each action’s geometric goal:

∃ ξ1, …, ξn :   ξi(0) = ξi−1(1),   ξi ⊂ Cfree,   ξi(1) ∈ G(ai)
(3.2)

Integrated task and motion planning (TAMP) solves the two together rather than in sequence (Garrett et al., 2021), because the layers constrain each other. A task plan may require a grasp that is geometrically impossible; a motion plan may reveal that only the top shelf is reachable, which changes the task plan.

“Put the cup in the cupboard” can fail at three layers. Task: the cupboard door is closed, and a domain that omits open(door) produces a valid three-step plan that cannot be executed — a model error, not a search error. Motion: the cup is behind a chair, so no collision-free path exists; the planner must change the grasp or request a task-level action. Control: the cup is heavy and the gripper slips; the plan was fine, the execution was not.

Caveat · hierarchy is a design choice

Not every agent has three explicit layers. An end-to-end vision-language-action policy collapses them into one network. A learned options framework discovers temporally extended actions without a symbolic task planner. The hierarchy manages time scales and makes interfaces inspectable; it is not a claim that intelligence must be modular. Ask which layer owns which decision, and what happens when the layers disagree.

04 / Language agentsLanguage proposes; the world disposes.

A large language model is a prior over text. It knows that a fridge usually contains food and that “put the apple in the fridge” usually means opening the door. That prior supplies semantics, task decomposition, and common-sense orderings no hand-written domain is likely to contain. It is not a model of the current room, and it does not guarantee that a plan is executable. The four architectures below put it to work without pretending it is more than a prior.

Language as a scorer over skills: SayCan

SayCan (Ahn et al., arXiv:2204.01691) gives the robot a library of learned skills, each with a value function that estimates whether it will succeed from the current state. Given an instruction i and state s, a candidate skill a is scored by combining the language model’s probability of the skill text with the value function’s estimate:

score(a | i, s) ∝ pLM(a | i) · V(a | s)
(4.1)

The product is the grounding mechanism: the language model says what is semantically appropriate, the value function says what the robot can do now. “Clean the spill” may make find sponge and wipe table both plausible; if no sponge is visible, the value function for find sponge is low, and the agent looks for one first. Selection is greedy: pick the best skill, execute, re-score. There is no backtracking and no guarantee of reaching the goal. SayCan’s contribution is not a planner; it grounds a language prior in an affordance model at every step.

Language as an interleaved trace: ReAct

ReAct (Yao et al., arXiv:2210.03629) interleaves a reasoning trace with actions and observations: the model writes a thought, takes an action, reads the result, and writes the next thought. The thought is not a proof; it is a scratchpad that helps track the plan and handle exceptions. The loop is the point: the plan is revised in light of what is seen rather than committed in one shot. ReAct improves grounding when observations are informative; it does not guarantee that the thought is faithful, the action is safe, or the loop terminates. A model can loop or drift; the trace aids inspection but is not a certificate.

Language as a candidate generator: LLM-Modulo

LLM-Modulo (Kambhampati et al., arXiv:2402.01817) makes the division of labour explicit. The LLM generates candidate plans. External critics — a classical planner, a physics simulator, a formal verifier, a human — check them and return errors. The LLM repairs and tries again. The LLM is a generator, not the planner of record. The guarantee comes from the critics and the specification they encode. A weak critic accepts bad plans; a critic with the wrong specification rejects good ones; a slow critic makes the loop impractical.

The contrast: end-to-end vision-language-action

RT-2 (Brohan et al., arXiv:2307.15818) is the opposite design. A vision-language model is trained to emit action tokens directly from an image and an instruction; there is no explicit plan, map, or symbolic model. The strength is semantic generalisation: web knowledge transfers to unseen objects and instructions. The weakness is the one this lecture has been building: no explicit model means no counterfactual check, no plan to inspect, and no long-horizon guarantee.

Table 4.1 · Four language-conditioned architectures, their grounding mechanism, and their characteristic failure.
ArchitectureGrounding mechanismWhat it buysCharacteristic failure
SayCanLanguage score × value/affordanceFeasible skills; real-robot groundingGreedy; no backtracking; value errors
ReActThought, action, observationFeedback-driven revisionLoops; unfaithful traces; context drift
LLM-ModuloExternal critics check candidatesCheckable plans; explicit repairCritic quality; specification errors
VLA (RT-2)End-to-end instruction-to-action trainingSemantic generalisation; no hand-written modelNo plan; weak long-horizon precision
Caveat · language is a specification, not a goal

An instruction is a sentence, not a formal objective. “Clean the spill” does not say which tool, how clean, how fast, or what to avoid. The agent’s literal goal may be satisfied while the user’s goal is not. This is a specification problem, and no amount of perception or planning fixes it. The honest agent asks, exposes its interpretation, or is constrained by a policy that says what it may not do.

05 / Memory & toolsWhat the agent carries, and what it calls.

An agent’s model is not only the dynamics of the world. It is also what it remembers and what it can call. Memory extends the state beyond the current observation; tools extend the action space beyond the body. Both are interfaces, and both add failure surfaces.

CoALA (Sumers et al., arXiv:2309.02427) gives a useful taxonomy. Working memory is the current context: belief, plan, recent observations. Episodic memory stores past trajectories and outcomes. Semantic memory stores facts and maps. Procedural memory stores skills and policies. A retrieved episode is evidence; a retrieved fact is a claim; a retrieved skill is a policy with preconditions.

Table 5.1 · Four kinds of memory, and the failure each one invites.
MemoryContentsFailure
WorkingCurrent belief, plan, contextOverflow; a stale plan
EpisodicPast trajectories and outcomesWrong retrieval; overfitting to one episode
SemanticFacts, maps, attributesStale facts; conflicting sources
ProceduralSkills and policiesUnchecked preconditions; skill drift

A map is a special kind of semantic memory. An occupancy grid stores free, occupied, or unknown for each cell; a topological map stores places and connections; a metric map stores geometry. All three are beliefs, not the world. They can be stale (a door closed after the map was built), incomplete (a room not explored), or wrong (a reflective surface read as free). SLAM maintains map and pose together, with uncertainty. The lab’s agent is a miniature SLAM problem: it observes a window, writes the observation into an occupancy map, plans over the map, and discovers that the map was optimistic.

A tool is a callable action with a contract: what it needs, what it returns, what it costs, and how it fails. A planner, solver, code interpreter, database, search engine, perception API, and robot skill are tools. Delegation is powerful and dangerous for the same reason: the contract can be wrong. ReAct and Toolformer (Schick et al., arXiv:2302.04761) learn to interleave tool calls with reasoning; Voyager (Wang et al., arXiv:2305.16291) builds a library of verified skills. The library records what worked; it does not guarantee that a skill’s preconditions hold in a new scene. The guarantee moves only if the tool has one: a SAT solver’s “unsatisfiable” is checkable; a web search result is not. A tool call is evidence, not a fact.

Caveat · memory can poison the loop

A retrieved memory is a prior, not an observation. If the agent retrieves a stale fact (“the door is open”) and acts on it without checking, it has replaced perception with memory. If it copies a wrong precedent, it has replaced planning with imitation. Tag memory with provenance and time, and check it when the cost of being wrong is high.

06 / EvaluationMeasure the loop, not the answer.

An evaluation that reports only final success is like a compiler test that reports only “program crashed.” It tells you something is wrong, not where. The useful questions are about the loop: Did the agent perceive the right state? Did its model predict correctly? Did the planner find a valid plan? Did the controller execute it? Did the monitor catch the mismatch? A benchmark is a model of a task, and like any model it has a blind spot.

Measure task success (did the goal hold?), cost (steps, path length, energy, interventions), generalisation (unseen instructions, objects, layouts), robustness (noise, occlusion, dynamics, distribution shift), recovery (does the agent detect failure, and how fast?), calibration (does predicted success match observed success?), and safety. Compositional generalisation — a new combination of known parts — is the hard case, where a memorised policy and a compositional model diverge. Epistemic uncertainty is about the model or state, and more data can reduce it; aleatoric uncertainty is noise no data removes.

ALFWorld (Shridhar et al., arXiv:2010.03768) pairs a text-based abstract environment with a visual embodied one, so an agent can learn a plan in language and execute it in a simulated house. ALFRED (Shridhar et al., arXiv:1912.01734) tests long-horizon household tasks with grounded instructions. BabyAI (Chevalier-Boisvert et al., arXiv:1810.08272) is a grid-world benchmark for instruction following; its small state space makes it useful for sample efficiency and compositional language, not real perception. The Embodied Agent Interface (Li et al., arXiv:2410.07166) unifies a broad set of tasks and evaluates four LLM modules — goal interpretation, subgoal decomposition, action sequencing, and transition modelling — with fine-grained error metrics.

Table 6.1 · What four benchmarks test, and what they do not.
BenchmarkWhat it testsWhat it does not test
ALFWorldText-to-embodied transfer; abstract plans executed in a houseReal perception; contact-rich manipulation
ALFREDLong-horizon instruction following; sequencingOpen-world language; multi-agent interaction
BabyAICompositional instruction following; sample efficiencyReal images; continuous control
Embodied Agent InterfaceModule-level diagnosis: goal, subgoals, actions, transitionsA single deployment-ready score

A world model can be evaluated four ways, and they do not agree: prediction accuracy, planning performance when the model is used, sample efficiency, and robustness under distribution shift. MuZero’s lesson is that a model can be poor at reconstructing observations and excellent for planning; the inverse is also possible. If the only number in the paper is a reconstruction loss, the paper has not shown the model is useful for control.

Caveat · benchmarks are models too

A benchmark encodes assumptions about the object set, instruction grammar, layout, and success predicate. An agent can be good at the benchmark and bad in the world, or good in the world and bad at the benchmark’s interface. Ask not “what is the score?” but “what does this benchmark make easy, and what does it make invisible?”

07 / Failure modesWhere the loop breaks.

The loop gives a natural taxonomy. A failure can enter at perception (wrong state), at the model (wrong transition), at planning (wrong search or heuristic), at execution (wrong action), or at the specification (wrong objective). The categories are not exclusive — a perception error can look like a planning error — but naming the layer is the first step to fixing it.

Table 7.1 · A failure taxonomy for agents. The detection column is what a monitor can actually observe.
LayerFailureExampleDetectionMitigation
GroundingWrong referent, attribute, or relation“The red cup” when there are twoCross-view consistency; uncertaintyBetter perception; scene graph; asking
ModelMissing precondition; ignored effect; stale mapPlan walks through a locked doorPrediction error; contact; contradictionMonitor; replan; model revision
PlanningIncomplete search; bad heuristic; short horizonNo plan in a solvable problemPlan validator; completeness checkBetter search; longer horizon
ExecutionTracking error; slip; dynamic infeasibilityGripper drops the cupProprioception; force; visionCompliance; re-grasp; slower path
SpecificationAmbiguous or wrong goal“Clean the spill” with no standardHuman feedback; violationAsk; expose interpretation; limits

The valid plan for the wrong state. The planner is sound, the plan is optimal, and the robot fails because the initial state was misperceived. This is the most common failure in language-conditioned agents: the instruction and plan are coherent, but the referent is wrong. A plan validator cannot catch it, because the validator uses the same wrong initial state.

The valid plan for the wrong model. The state is right, the planner is right, and the model is wrong. The door is locked; the domain says it is open; the plan is valid in the domain. The fix is a monitor that compares predicted outcome with observation, and a replanning step that uses the corrected model. A planner cannot fix a model error by searching harder.

The right plan, badly executed. The plan is valid in a good-enough model, but the controller cannot track it, the object slips, or the robot is pushed. The fix is at the control layer: compliance, force limits, slower trajectories, and a monitor that detects deviation. Calling this a planning failure leads to the wrong fix.

The sharp version

A* is correct. The map is wrong. A planner’s soundness and completeness are properties of the planner relative to its model. They are not properties of the agent relative to the world. The gap between the two is the subject of monitoring, and it is where most deployed agent failures live.

Attribution is hard because the layers interact. A perception error changes the state; the plan is for the wrong state; the monitor sees a surprise that looks like a control error. Log the intermediate objects: belief, model, plan, predicted outcome, observation, mismatch. The Embodied Agent Interface makes this systematic; the lab’s trace is a small version.

08 / RecapWhat you should be able to do now.

01

Draw the loop — perceive, model, plan, act, monitor — and name the interface each stage uses.

02

Write the belief update and explain the predict and update steps.

03

Compare latent, symbolic, and hybrid models on state, transition, planning, data, inspectability, and guarantee.

04

Explain why a symbolic plan is only as good as its model, and name the frame, qualification, and ramification problems.

05

Separate task planning, motion planning, and control, and state the feasibility condition that links them.

06

Describe how SayCan, ReAct, and LLM-Modulo ground language, and what each does not guarantee.

07

Design an evaluation that separates grounding, model, planning, execution, and specification errors.

08

Run the lab in all three modes and explain each outcome from the loop, not from the score.

Exercises

  1. Trace the loop in the lab. Run all three modes. Record steps, replans, collisions, and nodes expanded. For each outcome, name the failed stage and the observation that would have caught it.
  2. Break the belief. In the door example, set P(“open” | open) = 0.6 and P(“open” | closed) = 0.4. Compute the posterior after one and two observations. How many are needed to reach 0.9?
  3. Write a domain with a missing precondition. Write a three-action PDDL domain for making tea. Remove the precondition that the kettle has water. Show the plan and the step at which execution fails.
  4. Compare two language agents. Take “put the apple in the fridge.” How would SayCan and ReAct each ground it, and which failure is each more likely to exhibit? What would you monitor?
  5. Design an evaluation. A home robot sometimes misidentifies objects and sometimes plans through closed doors. Design a test set and logging scheme that separates the two. What would you report besides success rate?
  6. Argue the other side. Make the strongest case that an end-to-end vision-language-action policy with enough data makes explicit world models unnecessary. What evidence would change your mind?

Check your understanding

Ten questions, with explanations. The score is not the point; the questions you cannot answer are the reading list.

Reading list

The first six are the spine of this lecture. The rest are the next layer.

World Models

Ha & Schmidhuber · 2018 · the VAE + RNN + controller lineage

Latent

Dream to Control: Learning Behaviors by Latent Imagination

Hafner et al. · 2019 · actor-critic in a learned latent world

Latent

Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Schrittwieser et al. · 2019 · value-equivalent models and MCTS

Latent

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Ahn et al. · 2022 · language scores times affordance values

Language

ReAct: Synergizing Reasoning and Acting in Language Models

Yao et al. · 2022 · thought, action, observation

Language
Evaluation

LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks

Kambhampati et al. · 2024 · generate, check, repair

Language

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

Shridhar et al. · 2020 · text plans, embodied execution

Benchmark

BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning

Chevalier-Boisvert et al. · 2018 · grid-world instructions

Benchmark

Cognitive Architectures for Language Agents

Sumers et al. · 2023 · working, episodic, semantic, procedural memory

Memory

LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

Liu et al. · 2023 · language to PDDL, planner to plan

Planning

A Formal Basis for the Heuristic Determination of Minimum Cost Paths

Hart, Nilsson & Raphael · IEEE Transactions on Systems Science and Cybernetics, 1968 · A*

Classic

STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving

Fikes & Nilsson · Artificial Intelligence, 1971 · operators, preconditions, effects

Classic
Source note · what is cited and what is drawn

The worked examples are toy calculations, not benchmark results. The systems are described from their papers; no numbers are invented. Diagrams are schematics; the lab is a deterministic grid world, not a benchmark. Classics cited in the text: McCarthy & Hayes, “Some Philosophical Problems from the Standpoint of Artificial Intelligence” (1969); McDermott et al., PDDL (1998); Garrett et al., “Integrated Task and Motion Planning” (2021).