A mobile manipulator is in a kitchen. The instruction is one sentence: “Put the apples in the fridge.” The camera sees a five-by-five window. The fridge is behind the robot. Nothing in the image contains the word fridge, the path to it, or the fact that the door is closed. Yet the sentence has to become a sequence of torques.
00 / The hookThe map is not the world.
Break the sentence into questions. Which apples? That is grounding: mapping words to objects, places, and relations. Where is the fridge? That is memory and a spatial model, because the fridge is outside the view. What happens if I open the door? That is a world model: a prediction of the consequences of an action. In what order? That is planning: search over a model for a sequence that reaches the goal. How do I move the arm? That is motion planning and control: continuous, collision-aware, feedback-driven. How do I know it worked? That is monitoring: comparing prediction with observation and repairing the plan or the model when they disagree.
Each question is a topic from earlier in the course. An embodied agent is what you get when you put them in a loop and require the loop to close in real time. The pieces are not new; the engineering problem is the seam between them.
An agent is a loop closed through a model. The model predicts what actions do; the loop compares predictions with observations and repairs whichever is wrong. A model without a loop is a prior. A loop without a model is a reflex. Most interesting failures happen at the seam between them.
“Agent” is overloaded. This lecture covers embodied decision makers (robots, simulators, grid worlds) and tool-using language agents (ReAct-style loops, LLM-Modulo). They share the loop and the model; they differ in what the model is made of and what counts as an action. This is about the composition, not a survey of RL.
| Stage | Question | What it produces | Course interface |
|---|---|---|---|
| Perceive | What is the world, given what I can see? | A belief over states and relations | Grounding; logic as data |
| Model | What would happen if I did a? | A prediction or a rollout | Representation; logic as architecture |
| Plan | Which sequence reaches the goal? | A task plan, a path, or a policy | Logic as search |
| Act | What command do I send now? | An action, trajectory, or torque | Control; architecture |
| Monitor | Did the world do what I predicted? | A mismatch signal and a repair | Logic as verifier |
The sections take the stages apart: §1 formalises the loop, §2 compares three world-model families, §3 separates the planning layers, §4 adds language, §5 memory and tools, §6 evaluation, and §7 failure modes. The lab lets you remove the model, or remove the loop, and watch.
01 / The agent loopA loop closed through a model.
The standard formalism is a partially observable Markov decision process (POMDP). A state s ∈ S is the part of the world that matters; an action a ∈ A changes it; an observation o ∈ O is what the agent senses. The agent does not know s. It maintains a belief bt(s) = P(st = s | o1:t, a1:t−1) and updates it after each action and observation. With a known transition model T(s′ | s, a) and observation model O(o | s′), the update is the Bayes filter:
The sum is the predict step: push the previous belief through the transition model. The multiplication by the observation likelihood is the update step: reweight the prediction by how well each state explains what was seen. η normalises. The equation is simple; the difficulty is that T and O are usually learned, approximate, and drifting.
A policy maps a belief (or a history) to an action. A model-based agent queries a model at decision time and evaluates counterfactuals; a reactive agent maps the current observation directly to an action. The distinction is not “has a model in its weights” versus “has none” — a reactive policy can encode a great deal of structure. It is whether the agent queries a model of consequences when it decides. That is the difference the lab makes visible.
- Perceive. A detector proposes objects and attributes; a symbolic layer asserts facts such as
on(cup, table). The output is a claim with uncertainty, not a fact. Treating it as ground truth and blaming the planner is the most common design error in agent systems. - Model. A map, simulator, learned dynamics model, PDDL domain, language prior, or mixture. It answers “what would happen if?” for one action, a rollout, or a distribution. A model that cannot answer counterfactuals is a representation, not a world model.
- Plan. Search or optimisation over the model for operators, a trajectory, a policy, or a contingency tree. Planning is the deliberate version; a learned policy is the compiled version. Only the first is inspectable at decision time.
- Act. Execute one action, a short prefix, or the whole plan. No feedback is open loop; one step and re-plan is closed loop. The controller closes the fastest loop.
- Monitor. Compare prediction with observation. A mismatch can trigger re-perception, re-planning, slower execution, or a model update. This is the inference step that keeps the loop honest.
belief ← prior
plan ← ∅
repeat
o ← sense()
belief ← update(belief, o) # grounding / perception
if plan is empty or inconsistent(belief, plan):
plan ← search(model(belief), goal) # task + motion planning
a ← first(plan)
execute(a) # controller closes the inner loop
predicted ← rollout(model(belief), a)
if disagree(predicted, o):
replan or revise(model) # monitor / verifier
until goal(belief) or budget exhausted
The pseudo-code hides the key engineering choice: the rates. A controller runs at hundreds of hertz; perception at tens; task planning at a fraction of a hertz; model learning offline. A planner cannot run at servo rate, and a torque loop cannot close at planning rate.
Suppose a robot needs to know whether a door is open. Its prior is even: P(open) = 0.5. Its sensor reports “open” with probability 0.9 when the door is open, and 0.1 when it is closed. After one “open,” the posterior is (0.5 × 0.9) / (0.5 × 0.9 + 0.5 × 0.1) = 0.9; a second independent “open” gives 0.81 / 0.82 ≈ 0.988. Two weak observations can beat one strong one only if errors are independent — usually false for a real sensor. If route A is 6 steps when the door is open and 20 when it is closed, while route B is always 8, then A costs 0.9 × 6 + 0.1 × 20 = 7.4 in expectation. The numbers are a toy; the point is that a plan is a bet on a model.
Equation 1.1 assumes T and O are known and stationary. In a real agent they are learned, approximate, and drifting. Repeated observations of the same misleading cue do not create information if the errors are correlated. The loop needs a monitor that detects “the world is not behaving as my model says,” not only a filter that updates within the model.
02 / World modelsLatent, symbolic, hybrid.
A world model predicts the consequences of actions and scores outcomes. It can be a map, a simulator, logical operators, a learned dynamics network, or a language prior. The three families are different answers to “what is the state, and how does it change?” Each answer buys something at a cost.
Latent world models
A latent model learns an encoder zt = encθ(ot), a transition zt+1 ~ pθ(zt+1 | zt, at), and heads for reward, value, or reconstruction. Planning happens in the latent space, not in pixels. The canonical factorisation is
Ha and Schmidhuber’s World Models (arXiv:1803.10122) compressed frames into a VAE latent, predicted the next latent with an MDN-RNN, and evolved a controller inside the learned dream. Dreamer (Hafner et al., arXiv:1912.01603) learns the model from pixels and trains an actor-critic by backpropagating through imagined latent rollouts. MuZero (Schrittwieser et al., arXiv:1911.08265) learns a model that predicts reward, policy, and value, not observations, and uses it inside Monte Carlo tree search.
Latent models scale to high-dimensional observations and learn without a hand-written abstraction. The cost is opacity and distribution dependence: a model can reconstruct beautiful images and still be useless for control. MuZero’s “value-equivalent” model is a reminder that prediction accuracy is not planning usefulness.
Compounding error
Model-based planning multiplies one-step errors over the horizon. Let the true transition f and the learned model f̂ both be L-Lipschitz, with one-step error bounded by ε. An open-loop rollout of length H satisfies
for L ≠ 1, and εH for L = 1. The bound telescopes: each step can add L times the previous error. It is worst-case and open-loop; a closed-loop agent re-observes and corrects. It explains why long-horizon planning is hard and why an optimiser can exploit the model’s optimistic mistakes.
Symbolic world models
A symbolic model represents state as facts and actions as operators with preconditions and effects. In the STRIPS convention (Fikes & Nilsson, 1971), an action a is applicable when its preconditions are a subset of the state; the successor deletes the delete-list and adds the add-list:
PDDL generalises this with typed objects, quantified goals, conditional effects, and durative actions (McDermott et al., 1998). A planner searches for a sequence that reaches the goal; the plan can be inspected, validated, repaired, and compared. The guarantee is conditional: a sound and complete planner achieves the goal in the model, not in the world.
Three classical problems mark the boundary. The frame problem (McCarthy & Hayes, 1969) is how to represent what does not change without enumerating every unaffected fact. The qualification problem is that preconditions are never exhaustively listable: a door can be locked, blocked, too heavy, or open inward. The ramification problem is that actions have indirect effects: moving a cup changes what is reachable, visible, and supporting something else. Explicitness is not completeness.
Hybrid world models
A hybrid model puts a learned representation and a symbolic structure together. Four patterns: object-centric perception plus symbolic relations (a detector builds a scene graph, as in NS-CL); a language model plus a classical planner (an LLM writes a PDDL problem, as in LLM+P, arXiv:2304.11477); learned skills plus a symbolic task layer (SayCan grounds a skill sequence in affordances); and learned dynamics plus symbolic constraints (a neural proposal checked by collision or kinematic constraints).
Hybrids relocate the interface problem. A detector labels a closed door as open; the fact is used; the plan is valid; the robot collides. Each component can be correct and the system still fails.
| Dimension | Latent | Symbolic | Hybrid |
|---|---|---|---|
| State | Continuous vector or distribution | Set of facts, typed objects | Objects, attributes, relations |
| Transition | Learned network; stochastic | Operators with preconditions and effects | Learned dynamics plus constraints |
| Planning | Rollouts, MPC, gradient-based | Search over operators; A*, PDDL | Task planner over skills; verifier |
| Data efficiency | Needs interaction data | Needs a written model | Needs both, usually less of each |
| Inspectability | Low; latent state is opaque | High; plan and operators are readable | Medium; the seam is the opaque part |
| Guarantee | Empirical, distribution-dependent | Soundness relative to the model | As good as the weakest component |
| Typical failure | Compounding error; model exploitation | Missing precondition; wrong initial state | Grounding error in a valid plan |
Model-free RL can be trained inside a model-based simulator; a model-based planner can use a model with no explicit reward. The useful distinction is whether the agent evaluates counterfactual action sequences at decision time. A policy that memorised a maze is not planning; a planner that queries a learned model is.
03 / Planning & controlThree layers, three time scales.
“Planning” is used for three problems with different state spaces, algorithms, success criteria, and failure modes. Confusing them produces bad evaluations: a system is said to “fail at planning” when the task plan was fine and the controller could not track it, or when the task plan was geometrically infeasible from the start.
| Layer | State space | Decision | Rate | Success | Typical failure |
|---|---|---|---|---|---|
| Task planning | Discrete facts and objects (state space) | Which operator next | Seconds to minutes | Goal formula satisfied | No plan; wrong model; wrong goal |
| Motion planning | Continuous configuration space | Collision-free path | 0.1–10 s | Path exists and is collision-free | Infeasible; narrow passage; dynamic obstacle |
| Control | Continuous state and commands | Torque or velocity command | 100–1000 Hz | Stable tracking; bounded error | Instability; saturation; slip |
Task planning searches over discrete operators. The input is a domain, an initial state, and a goal; the output is a sequence, a partial order, or a policy. Forward search with A* and a heuristic is the baseline; delete-relaxation heuristics such as hadd and hmax make it practical. A task plan can be validated without executing it: check each operator’s preconditions and the final goal. That is a real guarantee about the model. If the initial state says the door is open and it is closed, the validator still certifies the plan.
Motion planning searches a continuous configuration space for a path that avoids obstacles and satisfies kinematic constraints. Sampling-based planners (RRT, PRM) explore; trajectory optimisation refines. Both need a collision model, which is itself a world model. A path can be collision-free and dynamically infeasible: too fast for the motors, too tight for the gripper, or unreachable from the current base pose.
Control closes the fastest loop. A PID controller corrects tracking error; model predictive control (MPC) optimises a short horizon at each tick and executes the first action:
Receding-horizon execution is the loop: optimise, execute one step, observe, optimise again.
A task plan is feasible only if the motion layer can realise each abstract action. For a task plan σ = (a1, …, an) and trajectories ξi, the plan is realisable when the trajectories connect, stay collision-free, and end in each action’s geometric goal:
Integrated task and motion planning (TAMP) solves the two together rather than in sequence (Garrett et al., 2021), because the layers constrain each other. A task plan may require a grasp that is geometrically impossible; a motion plan may reveal that only the top shelf is reachable, which changes the task plan.
“Put the cup in the cupboard” can fail at three layers. Task: the cupboard door is closed, and a domain that omits open(door) produces a valid three-step plan that cannot be executed — a model error, not a search error. Motion: the cup is behind a chair, so no collision-free path exists; the planner must change the grasp or request a task-level action. Control: the cup is heavy and the gripper slips; the plan was fine, the execution was not.
Not every agent has three explicit layers. An end-to-end vision-language-action policy collapses them into one network. A learned options framework discovers temporally extended actions without a symbolic task planner. The hierarchy manages time scales and makes interfaces inspectable; it is not a claim that intelligence must be modular. Ask which layer owns which decision, and what happens when the layers disagree.
04 / Language agentsLanguage proposes; the world disposes.
A large language model is a prior over text. It knows that a fridge usually contains food and that “put the apple in the fridge” usually means opening the door. That prior supplies semantics, task decomposition, and common-sense orderings no hand-written domain is likely to contain. It is not a model of the current room, and it does not guarantee that a plan is executable. The four architectures below put it to work without pretending it is more than a prior.
Language as a scorer over skills: SayCan
SayCan (Ahn et al., arXiv:2204.01691) gives the robot a library of learned skills, each with a value function that estimates whether it will succeed from the current state. Given an instruction i and state s, a candidate skill a is scored by combining the language model’s probability of the skill text with the value function’s estimate:
The product is the grounding mechanism: the language model says what is semantically appropriate, the value function says what the robot can do now. “Clean the spill” may make find sponge and wipe table both plausible; if no sponge is visible, the value function for find sponge is low, and the agent looks for one first. Selection is greedy: pick the best skill, execute, re-score. There is no backtracking and no guarantee of reaching the goal. SayCan’s contribution is not a planner; it grounds a language prior in an affordance model at every step.
Language as an interleaved trace: ReAct
ReAct (Yao et al., arXiv:2210.03629) interleaves a reasoning trace with actions and observations: the model writes a thought, takes an action, reads the result, and writes the next thought. The thought is not a proof; it is a scratchpad that helps track the plan and handle exceptions. The loop is the point: the plan is revised in light of what is seen rather than committed in one shot. ReAct improves grounding when observations are informative; it does not guarantee that the thought is faithful, the action is safe, or the loop terminates. A model can loop or drift; the trace aids inspection but is not a certificate.
Language as a candidate generator: LLM-Modulo
LLM-Modulo (Kambhampati et al., arXiv:2402.01817) makes the division of labour explicit. The LLM generates candidate plans. External critics — a classical planner, a physics simulator, a formal verifier, a human — check them and return errors. The LLM repairs and tries again. The LLM is a generator, not the planner of record. The guarantee comes from the critics and the specification they encode. A weak critic accepts bad plans; a critic with the wrong specification rejects good ones; a slow critic makes the loop impractical.
The contrast: end-to-end vision-language-action
RT-2 (Brohan et al., arXiv:2307.15818) is the opposite design. A vision-language model is trained to emit action tokens directly from an image and an instruction; there is no explicit plan, map, or symbolic model. The strength is semantic generalisation: web knowledge transfers to unseen objects and instructions. The weakness is the one this lecture has been building: no explicit model means no counterfactual check, no plan to inspect, and no long-horizon guarantee.
| Architecture | Grounding mechanism | What it buys | Characteristic failure |
|---|---|---|---|
| SayCan | Language score × value/affordance | Feasible skills; real-robot grounding | Greedy; no backtracking; value errors |
| ReAct | Thought, action, observation | Feedback-driven revision | Loops; unfaithful traces; context drift |
| LLM-Modulo | External critics check candidates | Checkable plans; explicit repair | Critic quality; specification errors |
| VLA (RT-2) | End-to-end instruction-to-action training | Semantic generalisation; no hand-written model | No plan; weak long-horizon precision |
An instruction is a sentence, not a formal objective. “Clean the spill” does not say which tool, how clean, how fast, or what to avoid. The agent’s literal goal may be satisfied while the user’s goal is not. This is a specification problem, and no amount of perception or planning fixes it. The honest agent asks, exposes its interpretation, or is constrained by a policy that says what it may not do.
05 / Memory & toolsWhat the agent carries, and what it calls.
An agent’s model is not only the dynamics of the world. It is also what it remembers and what it can call. Memory extends the state beyond the current observation; tools extend the action space beyond the body. Both are interfaces, and both add failure surfaces.
CoALA (Sumers et al., arXiv:2309.02427) gives a useful taxonomy. Working memory is the current context: belief, plan, recent observations. Episodic memory stores past trajectories and outcomes. Semantic memory stores facts and maps. Procedural memory stores skills and policies. A retrieved episode is evidence; a retrieved fact is a claim; a retrieved skill is a policy with preconditions.
| Memory | Contents | Failure |
|---|---|---|
| Working | Current belief, plan, context | Overflow; a stale plan |
| Episodic | Past trajectories and outcomes | Wrong retrieval; overfitting to one episode |
| Semantic | Facts, maps, attributes | Stale facts; conflicting sources |
| Procedural | Skills and policies | Unchecked preconditions; skill drift |
A map is a special kind of semantic memory. An occupancy grid stores free, occupied, or unknown for each cell; a topological map stores places and connections; a metric map stores geometry. All three are beliefs, not the world. They can be stale (a door closed after the map was built), incomplete (a room not explored), or wrong (a reflective surface read as free). SLAM maintains map and pose together, with uncertainty. The lab’s agent is a miniature SLAM problem: it observes a window, writes the observation into an occupancy map, plans over the map, and discovers that the map was optimistic.
A tool is a callable action with a contract: what it needs, what it returns, what it costs, and how it fails. A planner, solver, code interpreter, database, search engine, perception API, and robot skill are tools. Delegation is powerful and dangerous for the same reason: the contract can be wrong. ReAct and Toolformer (Schick et al., arXiv:2302.04761) learn to interleave tool calls with reasoning; Voyager (Wang et al., arXiv:2305.16291) builds a library of verified skills. The library records what worked; it does not guarantee that a skill’s preconditions hold in a new scene. The guarantee moves only if the tool has one: a SAT solver’s “unsatisfiable” is checkable; a web search result is not. A tool call is evidence, not a fact.
A retrieved memory is a prior, not an observation. If the agent retrieves a stale fact (“the door is open”) and acts on it without checking, it has replaced perception with memory. If it copies a wrong precedent, it has replaced planning with imitation. Tag memory with provenance and time, and check it when the cost of being wrong is high.
06 / EvaluationMeasure the loop, not the answer.
An evaluation that reports only final success is like a compiler test that reports only “program crashed.” It tells you something is wrong, not where. The useful questions are about the loop: Did the agent perceive the right state? Did its model predict correctly? Did the planner find a valid plan? Did the controller execute it? Did the monitor catch the mismatch? A benchmark is a model of a task, and like any model it has a blind spot.
Measure task success (did the goal hold?), cost (steps, path length, energy, interventions), generalisation (unseen instructions, objects, layouts), robustness (noise, occlusion, dynamics, distribution shift), recovery (does the agent detect failure, and how fast?), calibration (does predicted success match observed success?), and safety. Compositional generalisation — a new combination of known parts — is the hard case, where a memorised policy and a compositional model diverge. Epistemic uncertainty is about the model or state, and more data can reduce it; aleatoric uncertainty is noise no data removes.
ALFWorld (Shridhar et al., arXiv:2010.03768) pairs a text-based abstract environment with a visual embodied one, so an agent can learn a plan in language and execute it in a simulated house. ALFRED (Shridhar et al., arXiv:1912.01734) tests long-horizon household tasks with grounded instructions. BabyAI (Chevalier-Boisvert et al., arXiv:1810.08272) is a grid-world benchmark for instruction following; its small state space makes it useful for sample efficiency and compositional language, not real perception. The Embodied Agent Interface (Li et al., arXiv:2410.07166) unifies a broad set of tasks and evaluates four LLM modules — goal interpretation, subgoal decomposition, action sequencing, and transition modelling — with fine-grained error metrics.
| Benchmark | What it tests | What it does not test |
|---|---|---|
| ALFWorld | Text-to-embodied transfer; abstract plans executed in a house | Real perception; contact-rich manipulation |
| ALFRED | Long-horizon instruction following; sequencing | Open-world language; multi-agent interaction |
| BabyAI | Compositional instruction following; sample efficiency | Real images; continuous control |
| Embodied Agent Interface | Module-level diagnosis: goal, subgoals, actions, transitions | A single deployment-ready score |
A world model can be evaluated four ways, and they do not agree: prediction accuracy, planning performance when the model is used, sample efficiency, and robustness under distribution shift. MuZero’s lesson is that a model can be poor at reconstructing observations and excellent for planning; the inverse is also possible. If the only number in the paper is a reconstruction loss, the paper has not shown the model is useful for control.
A benchmark encodes assumptions about the object set, instruction grammar, layout, and success predicate. An agent can be good at the benchmark and bad in the world, or good in the world and bad at the benchmark’s interface. Ask not “what is the score?” but “what does this benchmark make easy, and what does it make invisible?”
07 / Failure modesWhere the loop breaks.
The loop gives a natural taxonomy. A failure can enter at perception (wrong state), at the model (wrong transition), at planning (wrong search or heuristic), at execution (wrong action), or at the specification (wrong objective). The categories are not exclusive — a perception error can look like a planning error — but naming the layer is the first step to fixing it.
| Layer | Failure | Example | Detection | Mitigation |
|---|---|---|---|---|
| Grounding | Wrong referent, attribute, or relation | “The red cup” when there are two | Cross-view consistency; uncertainty | Better perception; scene graph; asking |
| Model | Missing precondition; ignored effect; stale map | Plan walks through a locked door | Prediction error; contact; contradiction | Monitor; replan; model revision |
| Planning | Incomplete search; bad heuristic; short horizon | No plan in a solvable problem | Plan validator; completeness check | Better search; longer horizon |
| Execution | Tracking error; slip; dynamic infeasibility | Gripper drops the cup | Proprioception; force; vision | Compliance; re-grasp; slower path |
| Specification | Ambiguous or wrong goal | “Clean the spill” with no standard | Human feedback; violation | Ask; expose interpretation; limits |
The valid plan for the wrong state. The planner is sound, the plan is optimal, and the robot fails because the initial state was misperceived. This is the most common failure in language-conditioned agents: the instruction and plan are coherent, but the referent is wrong. A plan validator cannot catch it, because the validator uses the same wrong initial state.
The valid plan for the wrong model. The state is right, the planner is right, and the model is wrong. The door is locked; the domain says it is open; the plan is valid in the domain. The fix is a monitor that compares predicted outcome with observation, and a replanning step that uses the corrected model. A planner cannot fix a model error by searching harder.
The right plan, badly executed. The plan is valid in a good-enough model, but the controller cannot track it, the object slips, or the robot is pushed. The fix is at the control layer: compliance, force limits, slower trajectories, and a monitor that detects deviation. Calling this a planning failure leads to the wrong fix.
A* is correct. The map is wrong. A planner’s soundness and completeness are properties of the planner relative to its model. They are not properties of the agent relative to the world. The gap between the two is the subject of monitoring, and it is where most deployed agent failures live.
Attribution is hard because the layers interact. A perception error changes the state; the plan is for the wrong state; the monitor sees a surprise that looks like a control error. Log the intermediate objects: belief, model, plan, predicted outcome, observation, mismatch. The Embodied Agent Interface makes this systematic; the lab’s trace is a small version.
08 / RecapWhat you should be able to do now.
Draw the loop — perceive, model, plan, act, monitor — and name the interface each stage uses.
Write the belief update and explain the predict and update steps.
Compare latent, symbolic, and hybrid models on state, transition, planning, data, inspectability, and guarantee.
Explain why a symbolic plan is only as good as its model, and name the frame, qualification, and ramification problems.
Separate task planning, motion planning, and control, and state the feasibility condition that links them.
Describe how SayCan, ReAct, and LLM-Modulo ground language, and what each does not guarantee.
Design an evaluation that separates grounding, model, planning, execution, and specification errors.
Run the lab in all three modes and explain each outcome from the loop, not from the score.
Exercises
- Trace the loop in the lab. Run all three modes. Record steps, replans, collisions, and nodes expanded. For each outcome, name the failed stage and the observation that would have caught it.
- Break the belief. In the door example, set
P(“open” | open) = 0.6andP(“open” | closed) = 0.4. Compute the posterior after one and two observations. How many are needed to reach 0.9? - Write a domain with a missing precondition. Write a three-action PDDL domain for making tea. Remove the precondition that the kettle has water. Show the plan and the step at which execution fails.
- Compare two language agents. Take “put the apple in the fridge.” How would SayCan and ReAct each ground it, and which failure is each more likely to exhibit? What would you monitor?
- Design an evaluation. A home robot sometimes misidentifies objects and sometimes plans through closed doors. Design a test set and logging scheme that separates the two. What would you report besides success rate?
- Argue the other side. Make the strongest case that an end-to-end vision-language-action policy with enough data makes explicit world models unnecessary. What evidence would change your mind?
Check your understanding
Ten questions, with explanations. The score is not the point; the questions you cannot answer are the reading list.
Reading list
The first six are the spine of this lecture. The rest are the next layer.
World Models
Ha & Schmidhuber · 2018 · the VAE + RNN + controller lineage
Dream to Control: Learning Behaviors by Latent Imagination
Hafner et al. · 2019 · actor-critic in a learned latent world
Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
Schrittwieser et al. · 2019 · value-equivalent models and MCTS
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn et al. · 2022 · language scores times affordance values
ReAct: Synergizing Reasoning and Acting in Language Models
Yao et al. · 2022 · thought, action, observation
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Li et al. · 2024 · module-level diagnosis
LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks
Kambhampati et al. · 2024 · generate, check, repair
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Shridhar et al. · 2020 · text plans, embodied execution
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
Chevalier-Boisvert et al. · 2018 · grid-world instructions
Cognitive Architectures for Language Agents
Sumers et al. · 2023 · working, episodic, semantic, procedural memory
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan et al. · 2023 · the end-to-end contrast
LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
Liu et al. · 2023 · language to PDDL, planner to plan
A Formal Basis for the Heuristic Determination of Minimum Cost Paths
Hart, Nilsson & Raphael · IEEE Transactions on Systems Science and Cybernetics, 1968 · A*
STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving
Fikes & Nilsson · Artificial Intelligence, 1971 · operators, preconditions, effects
The worked examples are toy calculations, not benchmark results. The systems are described from their papers; no numbers are invented. Diagrams are schematics; the lab is a deterministic grid world, not a benchmark. Classics cited in the text: McCarthy & Hayes, “Some Philosophical Problems from the Standpoint of Artificial Intelligence” (1969); McDermott et al., PDDL (1998); Garrett et al., “Integrated Task and Motion Planning” (2021).