A child hears “put the red cube in the bowl” and does it. A robot must parse the sentence, find the cube, notice the bowl is occupied, decide what to move first, check that the arm can reach, turn the plan into joint commands, and notice if the cube slips. One sentence; six models; six places to be wrong.
00 / The hookOne sentence, six models.
The scene is a table with a red cube, a red sphere, a blue cube inside a bowl, and a knife. A fixed-base manipulator with a parallel-jaw gripper and a fixed camera looks down at it. The instruction is one sentence. Nothing is exotic; this is the tabletop pick-and-place task that has been a benchmark for decades, and it is still hard because the sentence and the motion are written in different languages.
Take it apart. Which object is “the red cube”? That is grounding: mapping a referring expression to an object, with attributes, relations, and a frame of reference. What order? That is task planning: a sequence of symbolic actions whose preconditions hold and whose effects reach the goal. Can the arm get there? That is motion planning: a collision-free path through configuration space. How do the joints move? That is control: a feedback law that turns a trajectory into commands. Did it work? That is monitoring: comparing prediction with observation. Was it allowed? That is a specification and a shield: a check over the trace, not just the final state.
Each question has its own representation, time scale, and guarantee. Pixels become symbols in tens of milliseconds; symbols become actions in seconds; geometry becomes torques in milliseconds. A failure in one layer looks like a failure in another: the wrong object looks like bad planning, a dropped object looks like bad control, and a knife knocked over looks like nothing at all if you only measure the goal.
Language specifies; symbols plan; geometry refines; control executes; a monitor checks. A robot is not one model but a stack of models at different levels of abstraction. The interesting failures are at the seams, where one layer's output is another layer's assumption.
This lecture is about that stack: what each layer does, what it can promise, and where a symbolic component earns its place next to a learned one.
The running example is tabletop manipulation. Mobile robots add localization, mapping, and navigation; legged robots add gait and balance; humanoids add whole-body control. The layers grow and the physics gets harder, but the pattern is the same. The lecture uses the simplest instance so the seams are visible.
01 / The robot stackFive layers, one loop.
It helps to draw the stack as a pipeline, though it is not one. On the left are the instruction and the scene; on the right is a robot that moves. Between them, five layers answer five questions and expose interfaces to their neighbours. The interfaces are the interesting part: they are where representations change and where a type error becomes a robot error.
| Layer | Representation | Time scale | Question | Output | Guarantee | Typical failure |
|---|---|---|---|---|---|---|
| Grounding | Objects, attributes, relations | 30–300 ms | What is where? | Symbols bound to objects | Empirical (detector accuracy) | Wrong referent, missed object |
| Task planning | Facts and action schemas | 0.1–10 s | What order? | Ground action sequence | Sound and complete relative to the domain | Missing precondition, wrong effect |
| Motion planning | Configuration space and geometry | 0.1–10 s | Which path? | Collision-free trajectory | Probabilistic or resolution completeness | Collision, unreachable pose |
| Control | Joint states, forces, dynamics | 1–10 ms | What command now? | Torques or velocities | Local stability and tracking bounds | Slip, oscillation, tracking error |
| Monitor / shield | Trace and specification | Every step | Is this allowed? | Allow, refuse, stop | Relative to spec and model | Blind spot, wrong specification |
The loop is what makes the stack more than a waterfall. A monitor can invalidate a plan; a motion planner can report that a symbolic action is infeasible; grounding can be re-run; a controller can report a stall. Failures go up. A system that only flows left to right is open loop, which is a decision to trust every model at every seam.
Each layer should state its contract in one sentence. The task planner answers plan(s₀, G) → π or no plan. The motion planner answers feasible(a, s) → τ or no trajectory. The controller answers track(τ) → commands. The monitor answers check(trace, φ) → verdict. A component that cannot say what it returns, and what that return is relative to, is not a layer.
Nothing requires these layers to be separate programs. A VLA collapses several of them into one network, and a reinforcement-learning policy can collapse all of them into one mapping from observation to action. That is legitimate, but it changes the guarantees: the collapsed system's promise is empirical, not compositional. The stack is the reference against which the collapse is measured.
02 / Task, motion, controlThree layers, three guarantees.
The word “planning” covers three different problems. They are often drawn as one box, which is why robotics papers talk past each other. Each layer searches a different space, runs at a different rate, and can promise a different thing.
Task planning: a plan is a proof about a model
A task planner works over a discrete state: a set of ground facts such as on(red_cube, table) and hand_empty. An action schema has a precondition — the facts that must hold — and an effect — the facts it adds and deletes. Applying an action is set union and difference:
A plan is a sequence a₁, …, aₙ such that every precondition holds where the action is applied and the final state satisfies the goal G. This is STRIPS (Fikes & Nilsson, 1971); its modern descendant is PDDL (McDermott et al., 1998). The plan is a proof relative to the domain and the initial state, and nothing else.
The lab is a small instance. The red cube is on the table, the blue cube is in the bowl, and the hand is empty. The goal is on(red_cube, bowl). Picking the red cube first fails at placement, because the bowl is occupied. The shortest plan is four actions: pick the blue cube, place it on the table, pick the red cube, place it in the bowl. The first action is not the object named in the instruction.
If the planner is sound and complete, and the domain, initial state, and goal are correct, the plan reaches the goal in the model. It does not promise that the actions are physically executable, that the symbols refer to the right objects, or that the goal is what the human meant.
The price is complexity: propositional STRIPS planning is PSPACE-complete (Bylander, 1994), and PDDL's typing and numeric fluents do not make it easier. Practical planners use heuristics and satisficing search. A planner that returns no plan may have proved that none exists or may have run out of time; the interface should say which.
Motion planning: a path for a body
A motion planner works over the robot's configuration space C. The obstacle region is the set of configurations in collision; C_free is its complement. A path is a continuous curve from a start to a goal configuration that stays in free space:
Sampling planners such as PRM and RRT are probabilistically complete: if a path exists, they find one with probability one as samples grow. RRT* is asymptotically optimal. None of that says anything about the task: a collision-free path can still be a bad grasp or a trajectory the controller cannot track. Completeness is a property of the search, not of the task.
Control: a feedback law, not a plan
The controller runs at 1–10 ms per cycle. It receives a reference trajectory and the measured state and produces torques, velocities, or an impedance. PID, computed torque, and MPC are answers to one question: given the state and the reference, what should the actuators do now? Control theory gives local stability and tracking bounds under assumptions. It cannot give task success, because the task is not in its model.
The seam: task and motion planning
The hard part is the seam. A task planner may choose pick(red_cube) because the symbolic precondition holds, while the arm cannot reach the cube: the bowl is in the way, the grasp is blocked, or the cube is too close to the edge. The action is applicable in the model and infeasible in the geometry.
Task and motion planning (TAMP) interleaves the two searches. A symbolic action applies only if a feasible motion realizes it, and the motion's endpoint must satisfy the symbolic effect:
There are three couplings. Plan first, then check is fast but incomplete: an infeasible plan forces backtracking over plans. Motion first, then abstract loses symbolic completeness: a feasible motion may not be the action the task needs. Interleave is the TAMP approach: the task planner asks for a feasible refinement of each action, and geometric failures prune the symbolic search. It is more complete and much more expensive. Garrett et al. (2021) is the standard map of the area.
| Layer | Search space | Output | What it can promise | What it cannot promise |
|---|---|---|---|---|
| Task planning | Finite sets of facts | Action sequence | Goal reachability in the model | Geometric feasibility, correct grounding |
| Motion planning | Continuous configuration space | Collision-free path or trajectory | Completeness (probabilistic or resolution) | Task success, grasp quality, control tracking |
| Control | Continuous state and dynamics | Commands at a fixed rate | Local stability, tracking bounds | Task success, collision avoidance, safety |
The sentence to keep: a task plan is a proof about a model, a motion plan is a path for a body, and a controller is a feedback law for a body in a world. None of the three is a promise about the world. When a robot fails, ask which promise broke, and whether the model it was relative to was true.
In tabletop pick-and-place the task plan is often two to six actions and geometry dominates: reachability, collisions, grasp poses, placement stability. In long-horizon household or industrial tasks the task plan dominates and geometric failures are rare. The bottleneck moves with the task, which is why a system tuned for one regime can look incompetent in another.
03 / Language policiesLanguage proposes; the body disposes.
The stack above is a classical architecture. The last five years added another answer: train one large model on images, language, and robot actions, and let it map the instruction and the observation directly to an action. The answer is powerful and easy to over-read. This section separates three systems that are often lumped together.
SayCan: score skills by plausibility and affordance
SayCan (Ahn et al., 2022, arXiv:2204.01691) keeps a library of pretrained skills — “find the sponge”, “wipe the table”, “pick up the can” — and asks a large language model which skill is the most plausible next step. A language model alone will happily propose a skill the robot cannot perform. SayCan multiplies the language score by a learned value function that estimates whether the skill will succeed from the current state:
The product is the interface. The LLM supplies semantic plausibility; the value function supplies an affordance estimate: how likely the skill is to succeed from the current state. The paper's phrase is that the robot acts as the language model's “hands and eyes.” The loop re-scores after each skill, so a blocked step changes the next choice. It is grounding in the literal sense: language tied to a body and a scene.
What SayCan is not: a greedy policy over a fixed skill library, not a task planner with backtracking and not a completeness proof. It cannot invent a skill; it can only choose among the ones it has. The value functions are learned and empirical, so a wrong value can make a plausible plan fail. The strength is that every step is checked against the robot's actual capabilities, which is exactly what a language model lacks.
RT-2: actions as tokens
RT-2 (Brohan et al., 2023, arXiv:2307.15818) starts from a vision-language model trained on web data, co-fine-tunes it on robot trajectories and vision-language tasks, and represents each action as a text token. The action becomes part of the same sequence as the language answer, so one model can be trained with one objective:
This is a policy, not a planner. The model sees an image and an instruction and emits action tokens; it has no explicit state, operator schemas, or goal formula. The bet is that web-scale pretraining supplies semantic knowledge a robot trained only on trajectories would lack — what a sponge is for, that spilled liquid should be wiped, that a can is graspable. The paper reports improved generalization and emergent semantic reasoning; those are empirical claims on its evaluation set, not guarantees.
OpenVLA: an open 7B policy
OpenVLA (Kim et al., 2024, arXiv:2406.09246) is an open-weights version of the idea: a 7B-parameter VLA built on a Llama 2 language model with a visual encoder that fuses DINOv2 and SigLIP features, trained on 970k real-world demonstrations from Open X-Embodiment and predicting discretized actions. Its abstract reports that it outperforms the closed RT-2-X (55B) by 16.5% absolute task success across 29 tasks with seven times fewer parameters. That is a result on a particular evaluation, not a law.
OpenVLA matters because the weights are open and the model can be fine-tuned on a new robot or task with modest data, which changes the economics of VLA research. It does not change the structure: the model is still a conditional policy over actions. It does not know the preconditions of its own behavior, cannot explain a plan, and cannot certify that an action is safe.
| System | Input | Output | Symbolic content | Grounding | Guarantee |
|---|---|---|---|---|---|
| SayCan | Instruction, state, skill library | Next skill | Skill names and preconditions | Value functions (affordances) | Empirical; greedy, no completeness |
| RT-2 | Image, instruction | Action tokens | None explicit | Learned in the weights | Empirical |
| OpenVLA | Image, instruction | Discretized action | None explicit | Learned in the weights | Empirical |
What a VLA does and does not guarantee
A VLA is a function from an observation and an instruction to a distribution over actions, trained by behavior cloning on demonstrations. That definition is enough to state its limits:
- No task-success guarantee. It generalizes empirically; a new object, layout, or instruction composition can break it.
- No explicit state. History is compressed into context or weights. It cannot show the facts it believes or be queried for a precondition.
- No planner. It emits an action or a short action chunk. Long horizons need a policy that learned the whole sequence or an external planner.
- No safety guarantee. Safety is a property of the executed trace. A policy trained to imitate has no incentive to respect a specification it never saw.
- No calibration by default. A policy can be confidently wrong. Uncertainty, OOD detection, and calibration are separate components.
The sixth limit is the data: spurious correlations in the demonstrations — background, object position, camera viewpoint — become part of the policy. This is not an argument against VLAs. It is an argument for putting them in a stack where a task planner supplies long horizons, a shield supplies safety, and a monitor supplies recovery. Compositional generalization is the property most often assumed and least often measured.
RT-2 and OpenVLA differ in data, scale, action tokenization, and control interface. A result on one does not transfer automatically to another. The structural claims here — no explicit state, no planner, no safety guarantee — are about the interface, not a benchmark ranking. A particular VLA may have a planner, an action-chunking scheme, or a safety layer bolted on; then the guarantee belongs to that component, not to the base model.
04 / Grounding & affordancesWhat the words pick out, and what the body can do.
Grounding turns “the red cube” into red_cube_1. It is not a lookup. The expression is resolved against a scene of objects, attributes, relations, and a point of view. A detector or a scene graph supplies candidates; a parser supplies constraints; a matcher scores them. The symbol grounding problem (Harnad, 1990) is the philosophical version: a symbol defined only by other symbols never touches the world. In robotics the problem is immediate.
The equation hides the hard parts. The score is learned and can be wrong. Attributes can be ambiguous: a “red cube” may be orange under another light. Relations can be frame-dependent: “left of the bowl” depends on whether left is the viewer's or the robot's. And the expression can have more than one referent. A system that always returns the top-scoring object is guessing; a system that reports the top two and asks knows it is grounding.
Affordance: what the object permits
An affordance is what an object or environment permits an agent to do with it. The term comes from Gibson's ecological psychology (1979), where it is relational: the same surface is walkable for a human and not for a beetle. In robotics, an affordance is a predicate over an object, a robot, and a task: graspable with this gripper, reachable from this base, stable on this surface, openable with this force. It is not a label on the object; it is a feasibility estimate conditioned on the body.
The distinction is between reference and action. A vision-language model can know the word “bowl” and not know that this bowl is too deep for the gripper or full of a blue cube. A grasp model can predict a good grasp and not know that the object is the wrong one. The two models fail in different places, and a system needs both.
| Failure | Example | What the planner sees | How to expose it |
|---|---|---|---|
| Wrong object | Two red objects; the expression is ambiguous | A valid plan for the chosen one | Log the top-k candidates and the margin; ask when the margin is small |
| Wrong attribute | Shape labels swapped by the detector | A valid plan for the wrong shape | Compare the executed goal with the intended referent |
| Wrong relation | “Left of the bowl” in the wrong frame | A valid plan for the wrong side | Declare the frame; test from more than one viewpoint |
| Wrong affordance | The gripper cannot span the object | A plan that fails at execution | Predict grasp success; check reachability before planning |
| Missed object | Occlusion or open-vocabulary failure | No candidate, or a wrong one | Report “not found” rather than the nearest label |
The most important row is the second. A grounding error is not a planning error. If the detector says the red sphere is the red cube, then pick(red_sphere) and place(red_sphere, bowl) are a valid plan for the symbol the planner was given. The planner cannot detect the mistake, because the mistake happened before its world began. Only a monitor with a goal over the intended referent — or a human looking at the bowl — can catch it. The lab's wrong-grounding mode is exactly this case.
A grasp that works for one object may fail for another with the same label. Learned affordance models inherit the objects, grippers, and lighting of their training data. “Graspable” is not a property of the object alone; it is a property of the object, the hand, and the task, estimated from a sample. Treat it as a prediction, not a fact.
05 / Shielding & safetyWhere the verifier enters.
Safety is a property of a trace, not of a single action. “Do not grasp the knife” is a property of every state. “Keep the end effector 0.3 m from the human” is a property of every instant. “Never put a sharp object in a food container” is a property of the final state and every transition into it. Some are discrete and symbolic; some are continuous and metric. A specification language has to say which.
A shield sits between the policy or planner and the robot. Given the current state and a proposed action, it returns the allowed actions, or a corrected action. In the preemptive form it filters before execution; in the post-hoc form it monitors and corrects only on a violation. A verifier can also enter earlier, as a plan checker. Alshiekh et al. (2018, arXiv:1708.08611) synthesize a shield from a temporal-logic specification and show how to preserve the learner's convergence guarantees under stated conditions. The construction is a game against the environment; the shield is a winning strategy for safety.
The guarantee is conditional, and the conditions matter more than the formula. It holds relative to the specification φ, the abstraction that maps the world to the shield's state, the transition model δ, and the correctness of the shield. If the specification is wrong, the shield enforces the wrong thing. If the abstraction is wrong — if the “knife” symbol is grounded to the wrong object — the shield protects the wrong object. If the model is wrong, the guarantee is about a world that does not exist.
| Level | Mechanism | What it guarantees | Cost | Blind spot |
|---|---|---|---|---|
| Task | Plan validation; constraints on actions; a critic that rejects a plan | No plan containing a forbidden action is executed | Plan search, constraint solving | Wrong symbols, wrong goal |
| Motion | Collision checking; joint, speed, and force limits | Every trajectory stays inside the geometric limits | Collision checking, sampling | Task-level safety, dynamic obstacles |
| Runtime | Monitor the trace; stop or switch to a backup | Violations are detected and the system stops | Sensor coverage, latency | Specification gaps, detection delay |
| Policy | Action filter around a learned policy | No unsafe action is sent to the actuators | Shield synthesis, conservatism | Wrong abstraction, reward hacking |
The lab's knife instruction shows the clean case. The unshielded planner clears the bowl, picks up the knife, and places it in the bowl. The plan achieves the goal. With the shield on, pick(knife) violates “never grasp a sharp object”, and the shielded search returns no plan. The correct behavior is to refuse and ask, not to relax the specification. A shield that silently drops the goal to find a safe plan is not a shield; it is a different task.
The subtle case matters most for neurosymbolic systems: safety and goal correctness are orthogonal. If the grounding is wrong, the shield can still guarantee that no sharp object is grasped while the robot puts the sphere in the bowl and fails the instruction. If the shield is off, the robot can achieve the goal with the knife and violate safety. The lab produces both outcomes, and both are failures — different failures with different fixes.
A shield can block safe actions, be expensive to synthesize, be specified for one abstraction and fail when the abstraction changes, and cannot fix a wrong goal, a wrong grounding, or a broken sensor. The guarantee is only as good as the specification and the model beneath it. This is Lecture 11's lesson again: verification is relative, and the relativity is the hard part.
06 / EvaluationSuccess is not enough.
The standard metric for an embodied agent is task success: did the goal hold at the end? It is necessary and insufficient. A single success rate aggregates a pipeline, and cannot say which model failed. Two systems can post the same success rate while one misreads goals, one grounds the wrong object, one plans badly, and one drops objects. The number is the same; the engineering problem is not.
The Embodied Agent Interface (Li et al., 2024, arXiv:2410.07166) makes the aggregation problem explicit. It formalizes a broad set of embodied decision tasks and evaluates four LLM-based modules separately: goal interpretation, subgoal decomposition, action sequencing, and transition modelling. It reports fine-grained errors — hallucinated preconditions, affordance errors, planning errors — rather than one success rate. The contribution is diagnostic: a score says whether; a module metric says why.
Two benchmarks recur. ALFWorld (Shridhar et al., 2020, arXiv:2010.03768) aligns a text-based abstract environment with an embodied visual one, so an agent can learn a plan in text and execute it in a scene; it tests language understanding, planning, navigation, and scene understanding together. LIBERO (Liu et al., 2023, arXiv:2306.03310) is a lifelong robot-learning benchmark with 130 tasks in four suites, designed to test transfer; it is also used as a simulation suite for fine-tuning and evaluating VLAs. Each benchmark is a model of a task, with a blind spot.
| Metric | Diagnoses | Hides |
|---|---|---|
| Task success | Whether the goal held | Which module failed; whether safety held |
| Goal-interpretation accuracy | Whether the instruction was understood | Grounding and execution |
| Grounding precision / recall | Whether the referent was found | Whether the affordance was right |
| Plan validity | Whether the plan is correct in the model | Geometric feasibility |
| Feasibility rate | Whether symbolic actions refine to motions | Grasp quality, control tracking |
| Safety violations | Whether the trace stayed inside the specification | Specification gaps |
| Recovery rate | Whether the system detects and repairs failures | Failures it never detects |
| Calibration | Whether confidence matches success | What the model does not know it does not know |
Evaluation also has confounders a leaderboard hides: the distribution of initial states, instruction paraphrases, the object set, the number of trials, the simulator version, the definition of success. A high success rate on one benchmark can coexist with a low one on a slightly different version. Report the distribution, not just the mean; report the number of trials; report confidence intervals when the sample is small. A benchmark is a model, and a model is only as good as the question it is asked.
Simulation gives scale, reproducibility, and cheap iteration. It also gives a sim-to-real gap: physics, contact, friction, sensor noise, lighting, and object variety are simplified. A policy that succeeds in simulation has demonstrated something about the simulator. Real-robot evaluation is expensive, underpowered, and hard to reproduce; that is a reason to design it carefully, not a reason to skip it.
07 / Failure modesA diagnostic table.
The stack gives a natural taxonomy. A failure belongs to the first layer where the system's model diverges from the world, not to the last layer where the symptom appears.
| Symptom | Layer | Likely cause | Test |
|---|---|---|---|
| Picks the wrong object | Grounding | Ambiguous expression, attribute swap, wrong frame | Log top-k candidates and scores; ask on a small margin |
| Plan is valid but impossible | Task–motion seam | Missing geometric precondition | Call the feasibility checker; interleave task and motion |
| Arm collides | Motion / control | Stale obstacle map, tracking error | Collision check at execution; monitor the pose |
| Object slips | Control / affordance | Grasp force, friction, bad grasp pose | Force sensing; regrasp; predict grasp success |
| Unsafe action executed | Specification / monitor | No shield, wrong spec, blind spot | Runtime monitor; formal spec; test the monitor |
| Goal achieved, safety violated | Specification | Success metric ignores safety | Report safety separately; add a shield |
| Fails on a new instruction | Parse / grounding | Out-of-distribution language | Paraphrase tests; ask for clarification |
| Works in simulation, fails on the robot | Distribution shift | Visual and physical gap | Domain randomization; real calibration; held-out scenes |
The procedure is mechanical. Freeze the instruction and the initial state. Record the parse, the grounding candidates and scores, the plan and its preconditions, the feasibility checks, the executed actions, the observations, and the monitor's verdict. Find the first place where the recorded state disagrees with the world. That is the layer to fix. Fixing a later layer first is how a system accumulates patches that hide the original error.
Two lab cases make it concrete. In the first, the detector swaps the shape labels of the red cube and the red sphere. The parser, planner, motion planner, and controller are all right; the robot puts the sphere in the bowl and the goal over the cube is false. The first divergence is grounding. In the second, the instruction names the knife. The unshielded plan achieves the goal and violates the specification; the first divergence is the absence of a specification at planning time. With the shield on, the system refuses — not success, not failure, but a correct refusal.
A VLA is a policy, not a planner. A planner is a proof, not a promise. A shield is a promise about a model, not about the world.
Errors interact. A grounding error can cause a planning error, which can cause an execution error, which the monitor reports as a control failure. The first-divergence rule is a heuristic, not a proof. It works when the layers are instrumented and the models are logged; it fails when a single end-to-end network hides the intermediate state. That is one of the costs of collapsing the stack.
08 / Cases & recapFour systems, one tuple.
The lecture has described four ways to build a robot system. Read them along the same tuple used throughout the course: representation, inference, coupling, guarantee. The differences are not labels; they are different answers to the question “what can this system promise?”
| System | Representation | Inference | Coupling | Guarantee |
|---|---|---|---|---|
| SayCan | Skills, LLM scores, value functions | Greedy skill selection | Score product at each step | Empirical; no completeness |
| RT-2 | Images, language, action tokens | Feed-forward policy | End-to-end | Empirical |
| OpenVLA | Images, language, discretized actions | Feed-forward policy | End-to-end, fine-tunable | Empirical |
| TAMP + shield | Symbols, geometry, specification | Task and motion planning; verification | Interleaved plus runtime filter | Relative to model and spec, not the world |
What the table makes visible: the systems with the strongest guarantees have the most explicit structure, and the most engineering. The systems with the broadest generalization have the least explicit structure, and their failures are hardest to attribute. That trade is the course's subject, in one domain.
Name the five layers and the question each answers: grounding, task planning, motion planning, control, monitor.
Write an action schema as precondition plus effects, and explain why a plan is a proof relative to a model.
Separate task planning, motion planning, and control by search space, time scale, and guarantee.
State the TAMP feasibility condition, and why the task–motion seam is where plans become impossible.
Compare SayCan, RT-2, and OpenVLA: representation, grounding, guarantee.
State what a VLA does not guarantee, and why “no explicit state” is the structural reason.
Diagnose a grounding error: a plan valid for the wrong symbol, and the monitor that catches it.
Place a shield, state its guarantee relative to the spec and model, and explain why safety and goal correctness are orthogonal.
Exercises
- Write the domain. Write the STRIPS operators for
pickandplaceby hand, including preconditions and add/delete lists. Run the lab and compare your plan with the planner's. - Break the grounding. Switch to the swapped detector and run the red-cube instruction. At which layer does the first divergence occur? What would a monitor need to know to catch it before execution?
- Turn off the shield. Run the knife instruction with the shield off. The goal is achieved. Write the two-sentence safety report, and explain why “success” is the wrong word.
- Design a specification. Write three safety properties for a robot handing objects to a person: one discrete, one metric, one temporal. For each, say which layer could enforce it and what its blind spot would be.
- Read a VLA paper. Take RT-2 or OpenVLA and fill the tuple. Which of the five VLA limits in §3 does the paper address, and which does it leave open?
Check your understanding
Ten questions, with explanations. The score is not the point; the questions you cannot answer are the reading list.
Reading list
The first five are the spine of this lecture. The rest are the next layer.
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn et al. · 2022 · SayCan: language scores times value functions
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan et al. · 2023 · actions as text tokens
OpenVLA: An Open-Source Vision-Language-Action Model
Kim et al. · 2024 · 7B open weights, 970k demonstrations
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Li et al. · 2024 · four modules, fine-grained errors
Safe Reinforcement Learning via Shielding
Alshiekh et al. · AAAI 2018 · the shield construction
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Shridhar et al. · ICLR 2021 · text plans, embodied execution
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
Liu et al. · 2023 · 130 tasks, four suites
Code as Policies: Language Model Programs for Embodied Control
Liang et al. · 2023 · language programs as controllers
LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks
Kambhampati et al. · ICML 2024 · the verifier in the loop
Integrated Task and Motion Planning
Garrett et al. · Annual Review of Control, Robotics, and Autonomous Systems, 2021 · the TAMP map
Figures 1.1 and 3.1 are schematic redrawings made for this lecture, not reproductions. The citations identify the systems and claims; the drawings are ours. The only comparative performance figure quoted from a paper is OpenVLA's 16.5% absolute improvement over RT-2-X across 29 tasks, from the paper's abstract; it is attributed and is a result on that evaluation, not a general law. The tabletop, plan, and monitor are teaching examples.